3

I have multiple point gpkg files which I would like to use with GeoPandas, however, reading files into the script takes always very long when files are bigger > 150MB.

Generally I just read the data with:

import geopandas as gpd

gdf = gpd.read_file(r'path/to/file.gpkg')

However, this is always pretty slow (takes multiple minutes). Is there some other way to read the data, but still be able to work with dataframes?

Improve this question
7
  • 1
    150MiB of points is an awful lot of points, so it would be expected to take some time. Since you haven't indicated the type of device on which the data is loaded (HDD/SSD), or the actual number of features, or even an exact time measurement for loading, we have no way to tell if this is a problem.
    – Vince
    Commented Nov 2, 2023 at 14:45
  • 150MB is actually relatively good to handle dataset, it takes couple of minutes (probably 2min to read) . Maybe geopandas is not even the right way to process those data, but I may need to switch to a Geodatabase as Postgre with PostGIS. However, I thought I first ask if there is a trick on how to load those data faster and still being able to use methods of geopandas.
    – i.i.k.
    Commented Nov 2, 2023 at 15:05
  • 1
    You're loading a ton of data from disk into RAM. That takes time. There's no reason to believe it would be faster from PostgreSQL (though it would cost hours/days to set that up and find out).
    – Vince
    Commented Nov 2, 2023 at 15:54
  • What would be a good choice then to process this kind of modest-sized geodata? I thought a postgreSQL setup with postGIS would do a good help. However, I think in the longer term I have to switch to something different (I think at using pyspark?).
    – i.i.k.
    Commented Nov 2, 2023 at 16:51
  • 1
    Check the benchmarks in gdal.org/development/rfc/rfc86_column_oriented_api.html. There are differences in the speed between the formats, but the differences between the implementation are much bigger. Geopandas seems to be the slowest of all the tested methods (with geopackage 227 seconds vs 4.8 seconds for the fastest alternative).
    – user30184
    Commented Nov 4, 2023 at 9:22

1 Answer 1

Reset to default
13

Install and use the optional pyogrio I/O engine to read the data... that will be a lot faster. Adding the use_arrow=True parameter as well will give another big performance improvement, but then you'll also have to install the pyarrow library:

import geopandas as gpd

gdf = gpd.read_file(r'path/to/file.gpkg', engine='pyogrio', use_arrow=True)

You can also change the defaults globally

  • geopandas.options.io_engine = "pyogrio"
  • os.environ["PYOGRIO_USE_ARROW"] = "1"

Some timings using a 360 MB .gpkg with polygon data (on windows):

read_file took 0:07:15.306107 with fiona engine (= the current default)
read_file took 0:00:24.831681 with pyogrio engine
read_file took 0:00:02.830280 with pyogrio engine and use_arrow=True
Improve this answer
11
  • 1
    @user30184 What Pieter showed on how long it takes with Fiona engine corresponds exactly to my experience with reading gpkg with that size with the fiona engine. I don't know what it causes
    – i.i.k.
    Commented Nov 4, 2023 at 9:32
  • 1
    @user30184 I know there is a big performance difference between windows and linux... and I'm working on windows.
    – Pieter
    Commented Nov 4, 2023 at 9:55
  • 3
    pyogrio started as a proof of concept that was discussed in a fiona issue and the decision was made to keep it separate because it would impact the API too much... So they are aware of the faster ways to deal with this. The next major version of geopandas will switch to using pyogrio by default...
    – Pieter
    Commented Nov 4, 2023 at 10:06
  • 1
    I just used the pyogrio-engine with pyarrow as suggested from you, and wow, it is such a big diffference! This changes a lot for my daily work. Many thanks to @Pieter
    – i.i.k.
    Commented Nov 5, 2023 at 13:01
  • 2
    I've tested this answer on a shapefile (.shp) nearly 3 times the size of the initial question. I find very similar performance enhancements across much larger files, Pieter's answer should be used when operating with all big datasets
    – TornadoEric
    Commented May 20 at 17:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.