1

I have a set of ~100,000 polygons (.shp format). Using python and ogr I would like to dissolve them to one feature class to speed up other downstream operations. The command I am using is:

ogr2ogr -f "ESRI Shapefile" dissolved.shp input.shp -dialect sqlite -sql "select ST_Collect(Geometry),common_attribute from input GROUP BY common_attribute"

The command takes some time to run (now over an hour, likely significantly longer). Doing the dissolve in ArcGIS and grouping by the common field results in the desired output in just a few seconds. My questions are (1) Why is ArcGIS so much faster? (2) How can I speed up this operation

3

When you tell GDAL to use SQLite SQL dialect the most part of the work is done by Spatialite library. Couple of weeks ago another user reported that ST_Collect in Spatialite was pretty slow for any bigger tasks https://groups.google.com/forum/#!topic/spatialite-users/t3QrvA1qQyg.

Week after that Alessandro Furieri announced about a code optimization which made ST_Collect about 400 times faster https://groups.google.com/forum/#!topic/spatialite-users/drNS_tiquJc.

Unfortunately it will take some time before all GDAL users can enjoy this speedup. The fix must first go to released Spatialite version and GDAL must be build with that version. Those who can build programs can build Spatialite from the trunk and use that for building GDAL.

  • This is fascinating. I will mark this as answered but I want to ask first, do you have any ideas around how I could speed up the operation? Looks like using geopandas might be a good idea? – geoeye Nov 23 '15 at 21:03
  • I am not sure but perhaps you can do what you want also with ST_Union. Try and tell us. – user30184 Nov 24 '15 at 5:50
  • ST_Union is generally "orders of magnitude slower". See postgis.net/docs/ST_Union.html The operation above ended up taking 3.5 hours. I am testing again with ST_Union now. Will test also using geopandas and update the question with results! – geoeye Nov 24 '15 at 14:36
  • 1
    If you are open for using other tools I am rather sure that you can do your job in couple of minutes with OpenJUMP. Run it with 64-bit Java because OpenJUMP must do all the job in the memory. Tools-Analysis-Union/Dissolve/Merge is probably your tool with the Use an Attribute to Dissolve checked. – user30184 Nov 24 '15 at 14:55
  • Unfortunately I have no experience with OpenJUMP or programming in Java :-( Additionally this analysis will be embedded in a larger system, which is all written in Python. This is why I'm looking for a pythonic solution – geoeye Nov 24 '15 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.