1

In earlier versions of QGIS I have used the PyQGIS code below to add OSM data to an existing GeoPackage, as a new layer (osm_lines). However, in my current version (3.34.5), the process is failing. My computer has recently been rebuilt, so I don't have the details of which earlier QGIS versions it was successfully running in.

The source for the data is an OSM PBF extract of Australia, converted to Spatialite format.

The issue appears to be in the gdal:convertformat call (which uses ogr2ogr behind the scenes). The new layer does not get added to the GeoPackage.

extentStr = '237474.0,259028.0,6110102.0,6137634.0 [EPSG:28356]'
resultOSMLines = processing.run("native:extractbyextent", { 'CLIP' : True, 'EXTENT' : extentStr, 'INPUT' : 'spatialite://dbname=\'C:/path/to/osm.db\' table=\"lines\" (geometry)', 'OUTPUT' : 'TEMPORARY_OUTPUT' })['OUTPUT']
processing.run("gdal:convertformat", { 'INPUT' : resultOSMLines, 'OPTIONS' : '-update -nln osm_lines', 'OUTPUT' : 'C:/path/to/gpkg.gpkg' })
vlOSMLines = QgsVectorLayer('C:/path/to/gpkg.gpkg'+'|layername=osm_lines', 'osm_lines', 'ogr')
QgsProject.instance().addMapLayer(vlOSMLines)

When I run gdal:convertformat (GDAL->Vector Conversion->Convert Format) through the Processing Toolbox, I get the following errors:

GDAL command:
ogr2ogr -f "GPKG" -update -nln osm_lines E:/geodata/morton/morton-ettrema_south.gpkg C:/Users/Brennant/AppData/Local/Temp/processing_YwSQRf/d1946cc5e17b45f29ef211e7aba26dde/INPUT.gpkg INPUT
GDAL command output:
ERROR 1: Cannot find OGR field for Arrow array ogc_fid 
ERROR 1: WriteArrowBatch() failed
ERROR 1: Terminating translation prematurely after failed 
translation of layer INPUT (use -skipfailures to skip errors)
Process returned error code 1
Execution completed in 0.50 seconds
Results:
{'OUTPUT': 'E:/geodata/morton/morton-ettrema_south.gpkg'}

I can work around the issue in the PyQGIS code by replacing the processing call to gdal:convertformat with calls to QgsVectorFileWriter.writeAsVectorFormatV3

extentStr = '237474.0,259028.0,6110102.0,6137634.0 [EPSG:28356]'
resultOSMLines = processing.run("native:extractbyextent", { 'CLIP' : True, 'EXTENT' : extentStr, 'INPUT' : 'spatialite://dbname=\'C:/path/to/osm.db\' table=\"lines\" (geometry)', 'OUTPUT' : 'TEMPORARY_OUTPUT' })['OUTPUT']
optionsOSMLines = QgsVectorFileWriter.SaveVectorOptions()
optionsOSMLines.layerName = 'osm_lines'
optionsOSMLines.actionOnExistingFile = QgsVectorFileWriter.CreateOrOverwriteLayer
writer = QgsVectorFileWriter.writeAsVectorFormatV3(resultOSMLines, 'C:/path/to/gpkg.gpkg', QgsProject.instance().transformContext(), optionsOSMLines)
vlOSMLines = QgsVectorLayer('C:/path/to/gpkg.gpkg'+'|layername=osm_lines', 'osm_lines', 'ogr')
QgsProject.instance().addMapLayer(vlOSMLines)

However, I'm keen to understand what has suddenly gone wrong with my existing code, and how to fix it.

2
  • Try if it works after setting environmental variable ` OGR2OGR_USE_ARROW_API=NO`. I think it can also be set as config option in the code, but I do not know how.
    – user30184
    May 17 at 5:00
  • @user30184 - yes, that seems to fix the problem (gdal.org/programs/ogr2ogr.html#known-issues). If you want to put it as an answer I can upvote/accept. I'll have to test whether recent versions of QGIS/GDAL fix the problem. May 17 at 9:02

1 Answer 1

1

The error seems to be somehow related to the relatively new "Column-oriented read API for vector layers" https://gdal.org/development/rfc/rfc86_column_oriented_api.html#rfc-86

The ArrowArray interface is rather well tested but sometimes it may still give troubles. See for example https://github.com/OSGeo/gdal/issues/9636 The possibility of ArrorArray related troubles is documented in the ogr2ogr manual https://gdal.org/programs/ogr2ogr.html#known-issues

Starting with GDAL 3.8, ogr2ogr uses internally an Arrow array based API (cf RFC 86: Column-oriented read API for vector layers) for some source formats (in particular GeoPackage or FlatGeoBuf), and for the most basic types of operations, to improve performance. This substantial change in the ogr2ogr internal logic has required a number of fixes throughout the GDAL 3.8.x bugfix releases to fully stabilize it, and we believe most issues are resolved with GDAL 3.9. If you hit errors not met with earlier GDAL versions, you may specify --config OGR2OGR_USE_ARROW_API NO on the ogr2ogr command line to opt for the classic algorithm using an iterative feature based approach. If that flag is needed with GDAL >= 3.9, please file an issue on the GDAL issue tracker.

When ogr2ogr is used through QGIS it may not be possible to add --config OGR2OGR_USE_ARROW_API NO to the ogr2ogr commands that QGIS creates, but the same effect can be achieved by setting an environmental variable OGR2OGR_USE_ARROW_API=NO.

1
  • 2
    I can confirm that processing.run("gdal:convertformat", { 'INPUT' : resultOSMPoints, 'OPTIONS' : '-update -nln osm_points --config OGR2OGR_USE_ARROW_API NO', 'OUTPUT' : 'C:/path/to/gpkg.gpkg' }) works, as does using the same additional creation options in the ogr2ogr (Convert Format) processing tool in QGIS 3.34.5 May 17 at 11:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.