ogr2ogr import fails with "ERROR 1: GeoJSON object too complex"

Question

I'm trying to import a large (8.8GB) GeoJSON file to a Postgres 9.6 database with the following command:

ogr2ogr -f "PostgreSQL" -nln final_diffs PG:"host=localhost dbname=inspire user=me" diffs.geojson

But I get lots of warnings like this:

ERROR 1: GeoJSON object too complex
ERROR 1: GeoJSON object too complex
More than 1000 errors or warnings have been reported. No more will be reported from now.

And the process runs without errors, but at the end, nothing has been imported into the Postgres database.

Any ideas? I've not seen this warning before, and the only online reference I can find is in the GDAL source code.

All I get from ogrinfo is the above warnings, then:

Layer name: diffs
Geometry: Unknown (any)
Feature Count: 0
Layer SRS WKT:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0,
        AUTHORITY["EPSG","8901"]],
    UNIT["degree",0.0174532925199433,
        AUTHORITY["EPSG","9122"]],
    AUTHORITY["EPSG","4326"]]

Answer 1

If you have GDAL >= 3.02, then you can set environment variable OGR_GEOJSON_MAX_OBJ_SIZE (as mentioned here) and run ogr2ogr, for example:

• Windows: SET OGR_GEOJSON_MAX_OBJ_SIZE=500MB
• Linux: export OGR_GEOJSON_MAX_OBJ_SIZE=500MB

Default value of OGR_GEOJSON_MAX_OBJ_SIZE is 200MB.

Answer 2

Error comes from the GDAL GeoJSON driver. The source code is in https://github.com/OSGeo/gdal/blob/master/gdal/ogr/ogrsf_frmts/geojson/ogrgeojsonreader.cpp.

The max value is defined on line 50

const size_t MAX_OBJECT_SIZE = 200 * 1024 * 1024;

There are tooComplex checks in many places and that throws the errot that you saw

void OGRGeoJSONReaderStreamingParser::TooComplex()
{
    if( !ExceptionOccurred() )
        Exception("GeoJSON object too complex");
}

I do not know really what to do but I can suggest

• Send a link of your data to gdal-dev mailing list and ask them to evaluate if that GeoJSON dataset is unreasonable or if the 200 * 1024 * 1024 limit should be raised again. It has already been doubled a few months ago because of this ticket https://github.com/OSGeo/gdal/issues/807.
• Raise the value yourself; it would require to compile GDAL from the sources
• Write a mail to the data provider and ask them to analyze if they could make the offending geometry smaller or use some other format, perhaps GeoPackage.

Answer 3

Wilhelm Berg and I ran into this problem recently, and he found that using tippecanoe-json-tool to convert the large geojson to a line-delimited geojson made it digestible for ogr2ogr:

tippecanoe-json-tool my_large_file.geojson > my_large_file.ldjson

ogr2ogr can then read the .ldjson file using the new GeoJSONSeq driver, which should prevent OOM errors:

ogr2ogr -f GPKG my_large_file.gpkg my_large_file.ldjson

NB: This has been tested on a 1.5 GB geojson - haven't tried on anything as large as the OP's 8.8 GB file.