# GeoPandas sjoin speedup

I am using GeoPandas's sjoin function to join 2 dataframes: dataframeA has latitude and longitude information whereas dataframeB has polygon information. Number of rows in dataframeA may vary (~70M) but are the same for dataframeB (825k). I want to perform point in polygon operation and update dataframeA with information from dataframeB. Here is my code which works (rtree and libspatialindex has been installed):

dataframeA = gpd.GeoDataFrame(dataframeA,crs="EPSG:4326",geometry=gpd.points_from_xy(dataframeA.longitude, dataframeA.latitude))
dataframeA = gpd.sjoin(dataframeA, dataframeB, op='within', how='left')

Since the memory requirement for this task is very high, I chunk dataFrameA before sjoin and append the results from disk. This process has been working fine.

Environment: Python 3.6; Dask - for high performance cluster

Problem: For chunked dataframeA (~7-8M rows), it takes about 2-3 hrs. I know point in polygon is computationally expensive.

Is there a way to speed this up?

• Have you tried dividing the polygons into smaller pieces?
– Bera
Commented Aug 14, 2020 at 11:34
• No, I have not tried that as it would increase number of point in polygon computations. Have you experienced improved performance doing so? Commented Aug 14, 2020 at 11:57
• Indeed, complex polygons are quite costly for point-in-polygon (which is O(N^2) for #vertices). There exists an optimal "sweet spot" where number of polygons isn't too high, but number of vertices is low enough. You might also try inverting the query to search B,A instead of A,B (though the results can be slightly different) Commented Aug 14, 2020 at 12:03
• I will try this. Any pointers on efficiently dividing polygons? One challenge I can think of is that the polygons are of different sizes - city, town, village, mall, business center, standalone business, etc. Would you still advise dividing the polygons in this case? Commented Aug 14, 2020 at 12:08
• You can get significant speedup if you have GeoPandas 0.8 and pygeos installed (geopandas.org/install.html#using-the-optional-pygeos-dependency). Commented Aug 17, 2020 at 8:57