Reading GeoPackage file bigger than RAM size with GeoPandas?

Question

I have a gpkg file that's around 10GB, and 8GB of RAM. So, if I were to read my file:

import geopandas as gpd
fp = "/path/to/my/file.gpkg"
data = gpd.read_file(fp)
print(data.shape)

My computer would crash. Now, in Pandas, you can read a file by chunks (https://stackoverflow.com/questions/25962114/how-to-read-a-6-gb-csv-file-with-pandas). I haven't seen anything related to such capabilities in the geopandas (or fiona, used to read the files) documentation.

What would be my options here?

• I tested converting my gpkg file to csv file in QGIS. I could then read it using Pandas and work my way up from there. While this could be a solution, I'd rather avoid this if possible.

• I could technically manually "tile" my data in QGIS, and get say 4 gpkg files of 2.5GB, but I don't want to do this.

• I could (and will) upgrade my RAM, but the problem is still there if I have 16GB of RAM and a 20GB gpkg file, etc.

Answer 1

The fastest way to read chuncked from a large geopackage is (as far as I know) to filter on the FID using a SQL statement, as this is the primary key.

You can execute SQL statements using geopandas.read_file.

Some rough, untested code that shows the idea:

import geopandas as gpd

start_fid = 0
chunk_size = 100_000
while True:
    sql = f'''
        SELECT * FROM "file"
         WHERE fid >= {start_fid} AND fid < {start_fid + chunk_size}
    '''
    gdf = gpd.read_file("file.gpkg", sql=sql)
    if len(gdf) == 0:
        break

    start_fid += chunk_size

    ...