# Accelerating GeoPandas for selecting points inside polygon

I have one dataset with about 10 million points (lat, long). I would like to select the points that fall within a map.

I have this map as a shapely object (`.shp`). To do this, I transformed the localization of each point into a Point object using `points_from_xy`. So I transformed my DataFrame into a GeoDataFrame.

My map is in the form of a Multipolygon object. So I used `geometry.unary_union` to convert it into a unified polygon (I'm not sure if it is correct). Then I used the `within` method in Geopandas to selects the Points inside the map.

``````import pandas as pd
import geopandas as gpd
from tqdm import tqdm

points = gpd.GeoDataFrame(df, geometry=gpd.points_from_xy(df.longitude, df.latitude))

map = map.geometry.unary_union

within_points = [points.geometry[i].within(map) for i in tqdm(range(points.geometry.count()))]
within_points = points[within_points]
``````

My problem is that this process takes TOO long (about 5 days on my core i5 laptop). I would like to know if there is any way to accelerate it?

• How big is your `foo1.csv'`, 10 Gb? Have you seen these articles Geopandas Intersects Speed and GeoPandas performance: optimizing vectorized operations? – Taras Jan 5 at 19:14
• Hi @Taras! It's too big, is about 1GB. Are you saying that I can use "sjoin" instead of "within"? I think the problem is in foo.shp. The geometry (multipolygon) has many details. It is a map o a biome in Brazil. – Vahid Jan 5 at 19:19
• Exactly, I am pointing to `spatial-join` as was mentioned by @gene. Can you apply that option? – Taras Jan 5 at 19:24
• @Taras, well suggestion! I will try to use this future. I will inform you here, if I succeed. – Vahid Jan 5 at 19:54
• Thank you for endorsing my comment, but I would highly suggest you rewrite your code, expand it with some details/information and put it as a valid answer. Hence it could be much helpful first of all for you, and secondly, let others comprehend explicitly the difference between your initial code and final code. IMHO, you may also extend it with a sort of a processing time calculated for instance with `time` package. – Taras Jan 6 at 12:27

With the help of @Taras I could solve my efficiency problem.

As the first try, I thought maybe the `tqdm` library is causing this delay. So I modified my code to

``````within_points = [points.geometry[i].within(map) for i in range(points.geometry.count())]
``````

As I wasn't using `tqdm` to know the remaining time, I waited about one day to see if the process terminates but as it took too long I aborted it. Using `tqdm` the rate of calculation was about 9.3 it/s and took about 114 hour!

The successful try was to rewrite the code and using `geopandas.sjoin` method. Before using this method I had to check if both GeoDataFrames have the same coordinate reference system (CRS).

In my the `map.crs` returned `{'init': 'epsg:4674'}` but `points.crs` returned nothing. To fix this, I assigned the result of `map.crs` to `points.crs`. After that, I used `geopandas.sjoin` with `op='within``.

As a comment, there two other options for `op` in `sjoin`, `'contains'` e `'intersects'`.

The modified code was surprisingly rapid and finished in less than 5min.

``````import pandas as pd
import geopandas as gpd