3

I'm trying to parallelize with multiprocessing.Pool() some processes that involve queries of R-trees. In the non-parallel procedure, it works as expected.

But when I try to do it in parallel (even with an example of running only 1 process in 1 core) the same query returns empty results.

I'm really puzzled about this, it goes beyond any logic for my understanding.

EDIT

Here is the simplest example that I could reproduce:

from multiprocess.pool import Pool
from rtree import index
class Test(object):
    def __init__(self):
        self.idx = index.Index()
        left, bottom, right, top = (0.0, 0.0, 10.0, 10.0)
        self.idx.insert(0, (left, bottom, right, top))

    def test(self, a):
        print(list(self.idx.intersection((1.0, 1.0, 2.0, 2.0))))

b = Test()
p = Pool()
res = p.map(b.test, range(3))
b.test(1)

output:

[]
[]
[]
[0]
  • can you provide more info on your data, your query and your libraries? – RutgerH Dec 3 '18 at 12:49
  • The data are two shapefiles, one with smaller polygons another with bigger ones, and i just retrieve the small polygons in each bigger one. The query is just a simple list(idx.intersection(big_pol.bounds)), and I use Python's Rtree and multiprocessing.Pool(). The index is not passed as a parameter, so there is no problem with pickle. – ImanolUr Dec 3 '18 at 14:23
4

The problem is that rtree is a ctypes wrapper around libspatialindex, and pickle has no way of storing the pointer to the rtree in memory.

You could instead store the index to disk:

from multiprocess.pool import Pool
from rtree import index

class Test(object):
    def __init__(self):
        pass

    def test(self, a):
        self.idx = index.Index('./rtree.idx')
        print(list(self.idx.intersection((1.0, 1.0, 2.0, 2.0))),)

idx = index.Index('./rtree.idx')
left, bottom, right, top = (0.0, 0.0, 10.0, 10.0)
idx.insert(0, (left, bottom, right, top))
idx.close()

b = Test()
p = Pool()
res = p.map(b.test, range(10))
  • I'm sure this is the way to go, and your example works as expected, but I can't manage to make it work in my real case. When I try to open the dictionary i get this error: rtree.core.RTreeError: Error in "Index_Create": Spatial Index Error: InvalidPageException: Unknown page id 1 – ImanolUr Dec 10 '18 at 10:54
  • Or it opens but the bounds are completely wrong: <class 'list'>: [1.7976931348623157e+308, 1.7976931348623157e+308, -1.7976931348623157e+308, -1.7976931348623157e+308] I'm debugging trying to see what goes wrong, but I can't understand it. – ImanolUr Dec 10 '18 at 11:21
  • I bet it's because rtree isn't threadsafe: github.com/Toblerity/rtree/issues/53 – Marc Pfister Dec 10 '18 at 14:49
  • I tried also to create a new index in place everytime ( thinking that the overhead of computing it thousands of times would be less than being able to be computing 64 at the same time) but still, I get empty responses to queries. So it does not crash, but it doesnt work either. I guess I will have to just give up with this. – ImanolUr Dec 10 '18 at 16:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.