I hope this question is not too general. I'm dealing with a lot of data from vehicles (1.000.000 GPS points per day) and I want to snap the data to a big city's street network. The timely order is not so important, so I don't need corrective filtering using HMM like in other implementations. I just need to snap the point to the nearest road.
I want to use Spark (pySpark) for this Problem and have
- gps_df: lons and lats
- edges_df: road network as shapely LineStrings (each Road is a new line and loaded as WKT to the DataFrame)
My first try was using osmnx using the
get_nearest_edge function, which is way too slow for the amount of GPS points.
Now I want to do this completely in Spark to make it fast and running parallel, but don't know where to start. Do I need kdtree? Which algorithm to chose (should I use shapely's
nearest_point, which uses rtree according to another post, or kdtree which is faster, or completely build it myself)?