I am in a bit of a weird work environment. I'm a data-scientist type person who is now doing more and more GIS work on some very large datasets - spatial regression, some mapping, spatial correlation, etc.
My data-engineer team are quite old-school and so instead of querying our spatial data directly, they only give us access to pre-processed shapefiles (most of my organisation is on Esri) on a network drive, to avoid storing data locally. Because of that, I tend to work in Python and have little SQL experience.
Managing large datasets with these is a pain - I tend to iterate over them in GeoPandas, and do most of my data manipulation in Python, which seems slow. I suspect ideally I'd do my initial querying in a database. I'm trying to find ways of improving that, but have some limitations!
I suspect the ideal solution would be a PostGIS SQL server, but we've been told that's impossible for now. I've considered a few options below...
- Spin up a local PostGIS server on my machine?
- aggregate those files into a SQLite/Spatialite local database
- aggregate them into geopackage db
Do these make sense? Should I be looking at something else? Do any of those sound mad?