3

I am learning how to implement the multiprocessing with spatial data using the module multiprocessing.

I am trying to implement a spatial intersection between a polygon file and a grid file. I am reading the data with geopandas only reading input files took 3 minutes: the grid's .shp file is 806 MB.

I am trying to implement this code(it is working ):

    data = []
    for index, orig in grid.iterrows():  # iterate over row on a dataframe
        for index2, ref in polygons.iterrows():
            if ref['geometry'].intersects(orig['geometry']): # if the feature intersect each other
             grid_id = orig['Id']
             gid_id = ref['gid_id']
             gid_pk = ref['gid_pk']

    data.append({'geometry':ref['geometry'].intersection(orig['geometry']),'grid_id':grid_id,
                 'nuts3_id': gid_id, 'nuts3_pk': gid_pk})
    df = gpd.GeoDataFrame(data,columns=['geometry','grid_id', 'nuts3_id', 'nuts3_pk'])
    df.to_file('intersection.shp')

Multiprocessing code

def chunks(l, n):
    for i in range(0, len(l), n):
        # yield l.loc[i:i+n ,:]
        f = l.loc[i:i+n -1 ,:]
        return f

def intersect_geom(z, polygons, d):
    for index2, ref in polygons.iterrows():

        if ref['geometry'].intersects(z['geometry']): # if the feature intersect each other

            grid_id = z['Id']
            gid_id = ref['gid_id']
            gid_pk = ref['gid_pk']
            inter_geom = ({'geometry':ref['geometry'].intersection(z['geometry']),'grid_id':grid_id,
             'nuts3_id': gid_id, 'nuts3_pk': gid_pk})


def main():
    data = []

    man = mp.Manager()
    d = man.dict()
    split = chunks(grid, len(grid)//10)
    for pos, z in split.iterrows():
        p = mp.Process(target=intersect_geom, args=(z, polygons, d))
        p.start()
        data.append(p)
        #
        # # wait that all chunks are finished
        [j.join() for j in data]




if __name__ == '__main__':

    main()

I got this error back

 Traceback (most recent call last):
  File "test_2.py", line 70, in <module>
    main()
  File "test_2.py", line 57, in main
    p.start()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.6/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/lib/python3.6/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.6/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.6/multiprocessing/popen_fork.py", line 65, in _launch
    parent_r, child_w = os.pipe()
OSError: [Errno 24] Too many open files
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 24, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 35, in <module>
    apt_pkg.init_system()
apt_pkg.Error: E:Error reading the Tuple table

Original exception was:
Traceback (most recent call last):
  File "test_2.py", line 70, in <module>
    main()
  File "test_2.py", line 57, in main
    p.start()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/usr/lib/python3.6/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/lib/python3.6/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/usr/lib/python3.6/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/usr/lib/python3.6/multiprocessing/popen_fork.py", line 65, in _launch
    parent_r, child_w = os.pipe()
OSError: [Errno 24] Too many open files

How can I fix this error and make the code run on multiprocessing?

  • 1
    Try a mulitprocessing.Pool() to make sure you're not spawning too many processes at once – mikewatt Jan 22 at 18:39
  • 1
    I suspect you want to make your chunks function yield rather than return, so that it is iterable. – alphabetasoup Jan 22 at 19:51
  • @RichardLaw if I use yield rather than return I got back this error: AttributeError: 'generator' object has no attribute 'iterrows' . I cannot iterate over the sub dataframe – Glori P. Jan 23 at 9:04
  • You might also be interested in the discussion here: github.com/geopandas/geopandas/issues/837 (experimental code to do parallel spatial join) – joris Jan 23 at 12:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.