3

I have run into some severe performance degradation when upgrading an environment to GDAL3. I could track the issue down to fiona.transform, which is a lot slower (about 15 (!) times) now than it was with GDAL 2.4.

The issue can be illustrated using this line, which only transform one point (the actual script transform a geometry):

python -m timeit -s "from fiona.transform import transform" "transform('EPSG:31287', 'EPSG:4236', [419908], [333400])"

These are my performance measurements with the images from perrygeo/gdal-base and using fiona 1.8.13:

latest  Python 3.8.5 | GDAL 3.1.3 | GEOS 3.8.1 | PROJ 7.1.1 | 20 loops, best of 5: 20 msec per loop
20181219-6f5f6a29   Python 3.6.7 | GDAL 2.4.0 | GEOS 3.7.1 | PROJ 5.2.0 | 1000 loops, best of 3: 675 usec per loop
20181219-f379ec62   Python 3.6.7 | GDAL 2.4.0 | GEOS 3.7.1 | PROJ 5.2.0 | 1000 loops, best of 3: 705 usec per loop
20181221-40f73e30   Python 3.6.7 | GDAL 2.4.0 | GEOS 3.7.1 | PROJ 5.2.0 | 1000 loops, best of 3: 698 usec per loop
20181221-bc2d4bbd   Python 3.6.7 | GDAL 2.4.0 | GEOS 3.7.1 | PROJ 5.2.0 | 1000 loops, best of 3: 688 usec per loop
20181221-f7a0a299   Python 3.6.7 | GDAL 2.4.0 | GEOS 3.7.1 | PROJ 5.2.0 | 1000 loops, best of 3: 703 usec per loop
20190312-f69f8699   Python 3.6.8 | GDAL 2.4.0 | GEOS 3.7.1 | PROJ 6.0.0 | 1000 loops, best of 3: 634 usec per loop
20190322-800eed8a   Python 3.6.8 | GDAL 2.4.1 | GEOS 3.7.1 | PROJ 6.0.0 | 1000 loops, best of 3: 589 usec per loop
20190509-da2e635a   Python 3.6.8 | GDAL 3.0.0 | GEOS 3.7.2 | PROJ 6.0.0 | 100 loops, best of 3: 10.4 msec per loop
20191110-6cc84c7e   Python 3.6.9 | GDAL 3.0.2 | GEOS 3.8.0 | PROJ 6.2.1 | 100 loops, best of 3: 11.4 msec per loop
20200301-8437abbb   Python 3.8.2 | GDAL 3.0.4 | GEOS 3.8.0 | PROJ 7.0.0 | 20 loops, best of 5: 10.5 msec per loop
20200509-50546ca8   Python 3.8.2 | GDAL 3.1.0 | GEOS 3.8.1 | PROJ 7.0.1 | 20 loops, best of 5: 10.2 msec per loop
20200907-c7ec91bc   Python 3.8.5 | GDAL 3.1.3 | GEOS 3.8.1 | PROJ 7.1.1 | 20 loops, best of 5: 10.8 msec per loop

Once can clearly see that the line performs at ~0.7 msec before GDAL3 and beginning with GDAL3 the line takes >10 msec to finish.

Does anyone have a hint, what could be the root of the issue and how it could be fixed?

6
  • 1
    Proj6 is slower to start than Proj4. This MapServer document mapserver.gis.umn.edu/ja/development/rfc/ms-rfc-126.html says that there is not much to do for single-time invocations. Perhaps there is something in Proj6 that you can cache so it starts faster, or some way to keep it up and waiting for the next input to convert.
    – user30184
    Oct 9, 2020 at 10:16
  • Thanks for your comment! That's what I thought at first, but interestingly there are two entries in my performance benchmark that have PROJ 6.0.0. Of course, GDAL 2.4 might not make use of new PROJ 6 functionality... Oct 9, 2020 at 10:19
  • GDAL 2.4 propably utilizes the deprecated proj_api.h header that is mentioned in osgeo.org/foundation-news/proj-7-0-0.
    – user30184
    Oct 9, 2020 at 10:29
  • The release notes of GDAL 3 state that "PROJ >= 6 is now a build requirement" (Source). This might be relevant too: github.com/Toblerity/Fiona/issues/799 ([discussion] optimizing transform_geom for repeated transformations) Oct 9, 2020 at 11:07
  • You could one transform to the setup statement, maybe that will catch the proj load. Oct 9, 2020 at 11:29

2 Answers 2

3

I would recommend using pyproj as it has dealt with this issue already: https://pyproj4.github.io/pyproj/stable/advanced_examples.html#optimize-transformations

The creation of the transformer has more overhead in PROJ 6+. That is why pyproj added the Transformer class. See: https://github.com/pyproj4/pyproj/issues/187

2

Indeed, as @snowman2 points out, using pyproj fixes the performance issue. The relevant command would look like this (for more complex geometries use shapely.ops.transform):

python -m timeit -s "from pyproj import Transformer" -s "transform = Transformer.from_crs(31287, 4236).transform" "transform(419908, 333400)"

It sets up a pyproj.Transformer that is being reused by the transformations.

The benchmark looks like this:

latest  Python 3.8.5 | GDAL 3.1.3 | GEOS 3.8.1 | PROJ 7.1.1 | 10000 loops, best of 5: 20.7 usec per loop
20181219-6f5f6a29   Python 3.6.7 | GDAL 2.4.0 | GEOS 3.7.1 | PROJ 5.2.0 | 10000 loops, best of 3: 29.8 usec per loop
20181219-f379ec62   Python 3.6.7 | GDAL 2.4.0 | GEOS 3.7.1 | PROJ 5.2.0 | 10000 loops, best of 3: 31.2 usec per loop
20181221-40f73e30   Python 3.6.7 | GDAL 2.4.0 | GEOS 3.7.1 | PROJ 5.2.0 | 10000 loops, best of 3: 56.9 usec per loop
20181221-bc2d4bbd   Python 3.6.7 | GDAL 2.4.0 | GEOS 3.7.1 | PROJ 5.2.0 | 10000 loops, best of 3: 33.3 usec per loop
20181221-f7a0a299   Python 3.6.7 | GDAL 2.4.0 | GEOS 3.7.1 | PROJ 5.2.0 | 10000 loops, best of 3: 56.4 usec per loop
20190312-f69f8699   Python 3.6.8 | GDAL 2.4.0 | GEOS 3.7.1 | PROJ 6.0.0 | 10000 loops, best of 3: 24.7 usec per loop
20190322-800eed8a   Python 3.6.8 | GDAL 2.4.1 | GEOS 3.7.1 | PROJ 6.0.0 | 10000 loops, best of 3: 83.9 usec per loop
20190509-da2e635a   Python 3.6.8 | GDAL 3.0.0 | GEOS 3.7.2 | PROJ 6.0.0 | 10000 loops, best of 3: 43.7 usec per loop
20191110-6cc84c7e   Python 3.6.9 | GDAL 3.0.2 | GEOS 3.8.0 | PROJ 6.2.1 | 10000 loops, best of 3: 58.6 usec per loop
20200301-8437abbb   Python 3.8.2 | GDAL 3.0.4 | GEOS 3.8.0 | PROJ 7.0.0 | 10000 loops, best of 5: 12 usec per loop
20200509-50546ca8   Python 3.8.2 | GDAL 3.1.0 | GEOS 3.8.1 | PROJ 7.0.1 | 20000 loops, best of 5: 10.5 usec per loop
20200907-c7ec91bc   Python 3.8.5 | GDAL 3.1.3 | GEOS 3.8.1 | PROJ 7.1.1 | 20000 loops, best of 5: 11.2 usec per loop

PS.: This is a thousand (!) times faster in GDAL 3/PROJ 7 than the fiona approach from the question.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.