3

I have a line delimited GeoJSON file with around 600k features.

Standard processing of it takes ~53 seconds:

ogr2ogr -f "Esri Shapefile" ./out ./in.geojson

Another approach I tried out was using gnu parallel:

concurrent_ogr() {
   ogr2ogr -f "Esri Shapefile" -sql "SELECT * FROM in_layer LIMIT 100000 OFFSET $1" ./out_$1 ./in.geojson
}

cat offsets.txt | parallel concurrent_ogr {}

where offsets.txt looks like

0
100000
200000
300000
400000
500000

This takes the same amount of time as the standard one (actually a little more).

I have 16 threads. The former query uses 1 at ~100%, while the latter uses 6 at ~100%.

Is it expected that the concurrent one would take as long as the non-concurrent one? Is there a way that the concurrent query could be made more performant?

1 Answer 1

5

You can't skip ahead in a GeoJSON file so each thread has to read up to the point you've asked it to start processing, discarding the features as it goes. This is one of the things that makes GeoJSON a bad interchange format.

5
  • I would clarify "bad interchange format if random access to data is desired". That makes GeoJSON also bad format for services which benefit from index based access to small subsets of data - WMS, WFS, OGC API Features etc.
    – user30184
    Mar 19, 2022 at 10:05
  • Or in fact almost any real world use, especially if you want projected data
    – Ian Turton
    Mar 19, 2022 at 13:34
  • Thanks! Are you familiar with any resources that describe formats that are good for random access?
    – Avocado
    Mar 19, 2022 at 17:44
  • Geopackage is good
    – Ian Turton
    Mar 19, 2022 at 18:42
  • Notice also that OFFSET...LIMIT is inefficient ivopereira.net/efficient-pagination-dont-use-offset-limit.
    – user30184
    Mar 20, 2022 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.