2

I have a GeoDataFrame with 1,2M records, and want to write it into a GeoPackage file. Currently, it takes >20 minutes, and, as I calculated, average write speed is 100 recs/sec.

As I read in the comments, the problem is that Fiona frequently locks and unlocks the sqlite database of GeoPackage.

Is there a way to improve write performance?

  • building Fiona on the machine?
  • opening the file in a specific mode?
  • setting environment parameters for Fiona & GDAL?

QGIS writes the same amount of records about 10 times faster, I'd be happy to see a similar improvement.

My OS is Ubuntu 18.04, but I also have a 18.10 one.

  • What version of GeoPandas and Fiona do you use? As I understand, you opened issues in both those packages last year that led to improvements in their latest versions. – joris Nov 15 '18 at 21:14
  • Fiona 1.8.0, Geopandas 0.4.0 – culebrón Nov 16 '18 at 8:02
  • 1
    Yes, in the meantime I checked with a test dataframe: it is now much faster than with GeoPandas 0.3 / Fiona 1.7, but still way slower than QGIS. Worth a further investigation. – joris Nov 16 '18 at 8:49
  • 2
    I opened new issues github.com/geopandas/geopandas/issues/863 and github.com/Toblerity/Fiona/issues/685 to track this. Certainly on the GeoPandas side it should be possible to improve it significantly. – joris Nov 17 '18 at 17:20
  • On the geopandas side, a significant performance improvement has been merged (github.com/geopandas/geopandas/pull/864), which will end up in the upcoming release. – joris Dec 9 '18 at 8:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.