1

I have many large gpkg.zip files that I would like to read directly into Python (to then upload to BigQuery). I can unzip each, read with geopandas, then upload -- however, since unzipping triples the space the file takes up, it starts to really clog up my computer's relatively small hard drive.

Is there a way to read a compressed file in directly?

1
  • 3
    Looked at stackoverflow.com/questions/26574407/… and seems not possible. Why don't you do sequentially, unzip file1, read file1, upload file1 content and delete unzipped file? Storage required on your computer = same size as all your zip + 3 times the size or your current zip.
    – ThomasG77
    Mar 12 at 17:33
2

You can use the GDAL /vsizip system.

I have this geopackage with one layer:

$ ogrinfo ottawa.gpkg 
INFO: Open of `ottawa.gpkg'
      using driver `GPKG' successful.
1: ottawa (Multi Polygon)

And I can create a zip file containing just it:

$ zip ottawa.gpkg.zip ottawa.gpkg 
  adding: ottawa.gpkg (deflated 38%)

And its nice and smaller than the source:

$ ls -sh ottawa.gpkg*
872K ottawa.gpkg  544K ottawa.gpkg.zip

Then I can access that file with /vsizip with the zip file name and the name of the file in the zip file:

$ ogrinfo /vsizip/ottawa.gpkg.zip/ottawa.gpkg 
INFO: Open of `/vsizip/ottawa.gpkg.zip/ottawa.gpkg'
      using driver `GPKG' successful.
1: ottawa (Multi Polygon)

The downside is this will be slower.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.