I have a gpkg file that's around 10GB, and 8GB of RAM. So, if I were to read my file:
import geopandas as gpd fp = "/path/to/my/file.gpkg" data = gpd.read_file(fp) print(data.shape)
My computer would crash. Now, in Pandas, you can read a file by chunks (https://stackoverflow.com/questions/25962114/how-to-read-a-6-gb-csv-file-with-pandas). I haven't seen anything related to such capabilities in the geopandas (or fiona, used to read the files) documentation.
What would be my options here?
I tested converting my gpkg file to csv file in QGIS. I could then read it using Pandas and work my way up from there. While this could be a solution, I'd rather avoid this if possible.
I could technically manually "tile" my data in QGIS, and get say 4 gpkg files of 2.5GB, but I don't want to do this.
I could (and will) upgrade my RAM, but the problem is still there if I have 16GB of RAM and a 20GB gpkg file, etc.