I have one dataset with about 10 million points (lat, long). I would like to select the points that fall within a map.
I have this map as a shapely object (.shp
). To do this, I transformed the localization of each point into a Point object using points_from_xy
. So I transformed my DataFrame into a GeoDataFrame.
My map is in the form of a Multipolygon object. So I used geometry.unary_union
to convert it into a unified polygon (I'm not sure if it is correct). Then I used the within
method in Geopandas to selects the Points inside the map.
import pandas as pd
import geopandas as gpd
from tqdm import tqdm
map = gpd.read_file('foo.shp')
df = pd.read_csv('foo1.csv')
points = gpd.GeoDataFrame(df, geometry=gpd.points_from_xy(df.longitude, df.latitude))
map = map.geometry.unary_union
within_points = [points.geometry[i].within(map) for i in tqdm(range(points.geometry.count()))]
within_points = points[within_points]
My problem is that this process takes TOO long (about 5 days on my core i5 laptop). I would like to know if there is any way to accelerate it?
foo1.csv'
, 10 Gb? Have you seen these articles Geopandas Intersects Speed and GeoPandas performance: optimizing vectorized operations?spatial-join
as was mentioned by @gene. Can you apply that option?time
package.