3

Is it possible to share rtree index in memory with multiple processes? I have succeeded to use rtree in multiprocessing environment with joblib, but the problem is that each process has its own copy of the index. With a fairly large multi gigabyte index and several processes it runs out of the memory.

I know that rtree is not thread-safe, but with only read operations would it be possible to store index in shared memory?

2
  • 2
    This seems more of a Python question than a GIS one. Stack Overflow is probably a more appropriate place to ask it.
    – Vince
    Nov 4 '21 at 18:27
  • 1
    This is more an rtree/libspatialindex specific question than a general python question and as such is on topic and should be left open.
    – user2856
    Nov 5 '21 at 7:51
7

One of the authors says no, read operations are not thread safe either.

However you could explore using a lock to prevent reads from happening simultaneously. Or running a separate process/thread whose sole job is to tend to the index, and querying it by passing data through queues.

If most of your code's execution time is spent doing queries then this would probably hurt performance too much to be feasible, but if hitting the index is just a small part of a larger processing pipeline then it could work out.

"Multi gigabyte index" also suggests that you might be storing [a copy of] the query-able data in the index itself. If your goal is to simply reduce the memory footprint then you could perhaps leave obj set to None and instead use the returned ID to look up the data which lives in shared memory, while continuing to make a copy of the index for each separate process.

1
  • Thanks for the suggestions! It had not occured to me that I could shave off bulk of the index size by just storing indexes to the objects. Nov 5 '21 at 21:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.