GeoDataFrame.overlay parallelization with multiprocessing

Question

I'm trying to speed up the calculation of the intersection between two geodataframes with multiprocessing. I'm using GeoPandas with JupyterLab from Anaconda on Linux.

My multiprocessing code currently works but has no speed benefit. In System Monitor it looks like one process is run after the other and not in parallel.

My code with mutliprocessing looks like this:

workers.py:

import geopandas as gpd
import numpy as np

def overl_wa_grid1(chunkind):
    #load mask and chunk
    grid1 = gpd.read_file('grid1.gpkg', engine='pyogrio')
    chunk = gpd.read_file(f'chunks/to_intersec_{chunkind}.gpkg', engine='pyogrio')
    
    #perform overlay
    chunk.overlay(grid1, how='intersection').to_file(f'chunks/intersec_{chunkind}.gpkg', driver='GPKG',engine='pyogrio')
    return f'chunks/intersec_{chunkind}.gpkg'

grid_intersec.ipynb:

import geopandas as gpd
import time
import numpy as np
import multiprocessing as mp
import workers

a =time.time()
#create and save chunks
chunks = np.array_split(gpd.read_file('to_intersec.gpkg', engine='pyogrio'), 8)
for i in range(8):
    chunks[i].to_file(f'chunks/to_intersec_{i}.gpkg', driver='GPKG',engine='pyogrio')

#parallelization of overlay
if __name__ ==  '__main__': 
    with mp.Pool() as pool:
        intersec_strings = list(pool.imap_unordered(workers.overl_wa_grid1, list(range(len(chunks)))))
intersec_dfs = []
print('int')
#load intersected chunks, concatenate and save result
for i in intersec_strings:
    intersec_dfs.append(gpd.read_file(i, engine='pyogrio'))
WA_intersec = pd.concat(intersec_dfs)
print('conc')
WA_intersec.to_file('intersected.gpkg', driver='GPKG',engine='pyogrio')
print('save')
b=time.time()
print(b-a)

This takes between 80 and 120s and the CPU graph looks like this:

The code without multiprocessing looks like this:

#load data
WA = gpd.read_file('to_explode.gpkg', engine='pyogrio')
WA.drop(columns=['...'], inplace=True)
WA_expl= WA.explode(column = 'geometry',ignore_index=True)
WA_expl.to_file('to_intersec.gpkg', driver='GPKG',engine='pyogrio')

#do overlay and save data
a =time.time()
WA_intersec = WA_expl.overlay(grid1, how='intersection')
WA_intersec.to_file('intersected.gpkg', driver='GPKG',engine='pyogrio')
del WA
del WA_expl
del WA_intersec
b=time.time()
print(b-a)

This takes around 70-80s and the graph looks like this:

My multiprocessing test code looks like this:

def f(x):
    return x*x

if __name__ ==  '__main__': 
    with mp.Pool() as pool:
        sqlist= list(pool.imap_unordered(f, range(100000)))

print('end')

The graph looks like this:

So is multiprocessing just the wrong package because its not CPU bound? Or does .overlay() just not work right with multiprocessing? Or is my code just wrong?

---

In the end I used rasterio to rasterize the polygons on a 10m raster and to do the further processing. One could probably also repolygonize the resulting rasters if polygons are needed. I didn't need the precise polygon overlays/intersections, if you do, the effort of trying to figure out multiprocessing could be worth it.

Answer 1

At first glance the code looks OK I think. If the intersection is actually with a grid (one layer are just simple rectangles), it is indeed quite possible that the bottleneck is rather in the file I/O than in the CPU needs.

I'm not sure if this is on purpose, but reading the data in not included in the timing in the code without multiprocessing.

You could give geofileops a try. This is a library that also uses multiprocessing to speed up geospatial operations. If the main bottleneck is I/O rather than CPU it probably won't be a lot better, but it might be worth the try.

The following (untested) code should do something similar:

import geofileops as gfo

# Explode
gfo.copy_layer('to_explode.gpkg', 'to_intersec.gpkg', columns=[...], explodecollections=True)

a = time.time()
gfo.intersection('to_intersec.gpkg', 'grid1.gpkg', 'intersected.gpkg')
b = time.time()
print(b-a)

Disclaimer: I'm the developer of geofileops.