GDAL org2ogr ESRIJSON Feature Paging

Question

I'm trying to load some data from an Esri Feature Server URL into PostGIS, using ogr2ogr, but seem to be facing some issues with paging.

The below code should in theory load all the Statutory Main Rivers from the UK's Environment Agency into PostgreSQL.

ogr2ogr -f PostgreSQL PG:"[connection string]" ESRIJSON:"https://services1.arcgis.com/JZM7qJpmv7vJ0Hzx/arcgis/rest/services/Statutory_Main_River_Map/FeatureServer/0/query?where=1%3D1&outFields=*&f=pjson" -oo FEATURE_SERVER_PAGING=YES -nlt MULTILINESTRING -nln test_stat_rivers

It however only loads in 4,000 records (which is listed as the maxRecordCount). I've checked and supportsPagination is set to true for it too. I have tried for another Feature Layer too, owned by my own organisation, where it was limited to 2,000 records (it's respective maxRecordCount) despite supporting pagination, so it wouldn't appear to be an issue isolated to a single layer.

Am I missing something here or is there a limitation in ogr2ogr or the ArcGIS REST API for handling this?

Answer 1

Turns out this was a bug within GDAL, related to using the Pretty JSON formatting (f=pjson), which @user30184 was kind enough to fix over on GitHub. As he explains on there, a workaround (before the fix is in a released) is to use f=json instead of f=pjson.