I have a gpkg file that's around 10GB, and 8GB of RAM. So, if I were to read my file:

import geopandas as gpd
fp = "/path/to/my/file.gpkg"
data = gpd.read_file(fp)

My computer would crash. Now, in Pandas, you can read a file by chunks (https://stackoverflow.com/questions/25962114/how-to-read-a-6-gb-csv-file-with-pandas). I haven't seen anything related to such capabilities in the geopandas (or fiona, used to read the files) documentation.

What would be my options here?

  • I tested converting my gpkg file to csv file in QGIS. I could then read it using Pandas and work my way up from there. While this could be a solution, I'd rather avoid this if possible.

  • I could technically manually "tile" my data in QGIS, and get say 4 gpkg files of 2.5GB, but I don't want to do this.

  • I could (and will) upgrade my RAM, but the problem is still there if I have 16GB of RAM and a 20GB gpkg file, etc.

  • Have you actually tried it? All modern OSs are able to swap out to disk in case they run out of RAM, that means things goes slower, but it should work. – MortenSickel Nov 6 '19 at 6:27
  • I guess you can use SQL "SELECT" with GeoPandas and therefore also LIMIT and OFFSET sqlite.org/lang_select.html. – user30184 Nov 6 '19 at 7:45
  • you could consider loading your data into PostGIS do the analysis there. There you do not have any problems with space – Jakob Miksch Nov 6 '19 at 7:56
  • You can read different geographically bounded chunks using bbox parameter. It depends what you have to do with your gpkg. If you need to work with all features at once, you might want to look for a different solution, but @MortenSickel is right, try that before. – martinfleis Nov 6 '19 at 9:02
  • My goal was to upload to PostGIS through python. I guess I can use ogr2ogr instead but I would have liked a python way. – William Abma Nov 6 '19 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.