1

I have a gdb file that's 32GB and I'm tasked with extracting the attribute values from each of the layers and converting them to CSV for analysis. Here's my code:

import geopandas as gpd
import fiona as fiona

dataset_path = '...' # for example purposes left out the exact path

layers = fiona.listlayers(dataset_path)
for layername in layers:
    # according to docs settin ignore_geometry=True loads gpd file to pandas dataframe
    geodata = gpd.read_file(dataset_path, driver='fileGDB', layer=layername, ignore_geometry=True)
    geodata.to_csv(project_path + f'{layername}.csv')

This code works but there are some layers that are very large (~5000 columns x 40'000 rows) and for those, it throws MemoryError and everything on my computer crashes because the PC has ran out of RAM.

Is there anything I can do to optimize this besides trying to acquire a more powerful computer? (I have 16GB of RAM).

  • I already tried to load the file into QGIS and export it from there, but it throws a similar error and crashes anyway.
4

2 Answers 2

1

Success.

  • I used the ignore_fields argument in read_file to drop unnecessary columns reducing the size of the import. It's not a quick solution, but at least it does not cause a memory error.
  • I also tried the rows method. According to documentation, you can provide a slice() object meaning that looping through the dataset and reading it bit by bit was not a problem.
0

The rows parameter for geopandas.read_file(file_path, rows=10) was not working for me. There were no warnings or error messages when using rows, but the entire dataframe was being loaded instead of the subset.

It took me a while to figure out that the rows parameter was added in geopandas=0.7, and I was running geopandas=0.6.1. I had installed geopandas with conda install geopandas in a Python=3.8 environment containing default Anaconda packages. I had issues trying to upgrade geopandas so instead I created a new environment from scratch but this time installing geopandas with conda install geopandas=0.9.0. This resolved the rows parameter issue for me.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.