3

I am using GeoPandas's sjoin function to join 2 dataframes: dataframeA has latitude and longitude information whereas dataframeB has polygon information. Number of rows in dataframeA may vary (~70M) but are the same for dataframeB (825k). I want to perform point in polygon operation and update dataframeA with information from dataframeB. Here is my code which works (rtree and libspatialindex has been installed):

dataframeB = gpd.GeoDataFrame(dataFromReadCSV,crs="EPSG:4326",geometry=geometry)    
dataframeA = gpd.GeoDataFrame(dataframeA,crs="EPSG:4326",geometry=gpd.points_from_xy(dataframeA.longitude, dataframeA.latitude))
dataframeA = gpd.sjoin(dataframeA, dataframeB, op='within', how='left')

Since the memory requirement for this task is very high, I chunk dataFrameA before sjoin and append the results from disk. This process has been working fine.

Environment: Python 3.6; Dask - for high performance cluster

Problem: For chunked dataframeA (~7-8M rows), it takes about 2-3 hrs. I know point in polygon is computationally expensive.

Is there a way to speed this up?

Improve this question
9
  • 1
    Have you tried dividing the polygons into smaller pieces?
    – Bera
    Commented Aug 14, 2020 at 11:34
  • No, I have not tried that as it would increase number of point in polygon computations. Have you experienced improved performance doing so?
    – researcher
    Commented Aug 14, 2020 at 11:57
  • Indeed, complex polygons are quite costly for point-in-polygon (which is O(N^2) for #vertices). There exists an optimal "sweet spot" where number of polygons isn't too high, but number of vertices is low enough. You might also try inverting the query to search B,A instead of A,B (though the results can be slightly different)
    – Vince
    Commented Aug 14, 2020 at 12:03
  • I will try this. Any pointers on efficiently dividing polygons? One challenge I can think of is that the polygons are of different sizes - city, town, village, mall, business center, standalone business, etc. Would you still advise dividing the polygons in this case?
    – researcher
    Commented Aug 14, 2020 at 12:08
  • 1
    You can get significant speedup if you have GeoPandas 0.8 and pygeos installed (geopandas.org/install.html#using-the-optional-pygeos-dependency).
    – martinfleis
    Commented Aug 17, 2020 at 8:57

1 Answer 1

Reset to default
5

You can get significant speedup if you have GeoPandas 0.8 and PyGEOS installed (geopandas.org/install.html#using-the-optional-pygeos-dependency). PyGEOS uses vectorized numpy ufuncs and can be orders of magnitude faster than standard shapely.

Note that PyGEOS will be part of Shapely 2.0, so once that is released, installing PyGEOS separately will not be needed.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.