1

I have a few files with location information:

1) User Lat Long Pings in Parquet Files (can be converted to CSV) format - this contains userid, timestamp, lat and lon
2) US County/State Shapefile
3) Store Centroid Data - centroid of stores like Walmart

Our goal is to:
1) Add geo attributes like county and state to user pings from county shapefile 2) Identify users who visited the store

Challenges:
1) Scale - User data has 30 million rows per day (over 6 months)

Can you please guide me on which geospatial libraries to use in order to create an efficient workflow? I briefly explored Fiona, Shapely, Geopandas.

I need to import the CSVs, shapefiles; define projections; create buffers; intersect them to get attributes and then output them. And I need to be able to do this at scale

Please let me know if my question is too vague

closed as too broad by PolyGeo Nov 30 '15 at 20:26

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • Using your first choice library where did you get stuck on your first step? That is the kind of question suitable for our focussed Q&A format. – PolyGeo Nov 30 '15 at 20:36
  • Is parallel processing available to you? That changes the problem considerably. This problem might be better suited as a hadoop tagged question on stack overflow, as there are definitely some workflows where you can solve this directly in hadoop. The Parquet -> CSV workflow may actually be your biggest bottleneck, since the rest of your problem can be solved at scale with proper sharding at that step. – blord-castillo Nov 30 '15 at 21:04