I have a data frame that consists of 2 million rows (lat, lon) and I am trying to figure out if each of these points are in a boundary defined by a shapefile or not (intact I am looking at multiple shapefiles). Here is the code, I have used to run this, but it is taking me around 30+ hours to run it. I am positive that there might be a better/faster way to do this and I am doing something incorrect. I have also tried the approaches suggested here but to no improvement in performance (R-Tree) R-Tree Spatial Index, Aggregating Points
Here is the code that I am currently using: To upload the shapefiles, I'm using geopandas, shapley, and pandas. I have downloaded the shapefiles from: www.zillow.com/howto/api/neighborhood-boundaries.htm, geo.nyu.edu/catalog/nyu_2451_34154, and geo.nyu.edu/catalog/nyu_2451_34496
import fiona as fn import shapely as sh import geopandas as gpd from shapely.geometry import Point import pandas as pd nyc_boroughs_df = gpd.read_file('./data/shapefiles/NYshapefiles/nyc_boroughs/nybb-geo_json.json') nyc_neighborhood_df = gpd.read_file('./data/shapefiles/NYshapefiles/ZillowNeighborhoods-NY/ZillowNeighborhoods-NY.shp') nj_neighborhood_df = gpd.read_file('./data/shapefiles/NJshapefiles/ZillowNeighborhoods-NJ/ZillowNeighborhoods-NJ.shp') ct_neighborhood_df = gpd.read_file('./data/shapefiles/CTshapefiles/ZillowNeighborhoods-CT/ZillowNeighborhoods-CT.shp') nyc_counties_df = gpd.read_file('./data/shapefiles/NYshapefiles/nyc_metro_counties/nyu_2451_34496-geojson.json') def get_boro_city_county_name(row): lat = row.lat lon = row.lon bn_series = nyc_boroughs_df[nyc_boroughs_df.geometry.contains(Point(lon, lat))].BoroName if len(bn_series) == 0: bn_series = nyc_neighborhood_df[nyc_neighborhood_df.geometry.contains(Point(lon, lat))].CITY if len(bn_series) == 0: bn_series = nj_neighborhood_df[nj_neighborhood_df.geometry.contains(Point(lon, lat))].CITY if len(bn_series) == 0: bn_series = ct_neighborhood_df[ct_neighborhood_df.geometry.contains(Point(lon, lat))].CITY if len(bn_series) == 0: bn_series = nyc_counties_df[nyc_counties_df.geometry.contains(Point(lon, lat))].name if len(bn_series) > 0: return str(bn_series.iat) else: return ''
df is a dataframe of 2 mil rows (columns: lat, lon) and when I apply the function on each of the rows, it takes around 30 hours to run.
df['name'] = df.apply(lambda x: get_boro_city_county_name, axis=1)