We have many datasets, each containing about 150'000 polygons. We would like to dissolve each dataset down to about 2500 polygons, based on a given attribute.

We've successfully done this using the ArcGIS Dissolve tool, but we would like to move towards an open-source solution. The problem is that every open source solution we have tried so far is incredibly slow. It takes ArcGIS about one to two minutes to dissolve one dataset. With other methods, it takes at least one hour, at which point I stop it since my guess is it has crashed completely due to memory issues.

These are the methods we have tried so far:

  1. QGIS Dissolve tool (NOT the Saga or GRASS tools). I use QGIS 2.18, would QGIS3 make a big difference?
  2. GeoPandas/Shapely.
  3. ogr2ogr (as described here http://darrencope.com/2015/01/23/dissolve-shapefiles-using-ogr/)
  4. We would gladly try other tools such as GRASS, but we require this to be able to run on a machine for which we do not have admin rights. This means no installations or sudo commands.

Are there are there other tools or methods we can try? Are there some "tricks" to try out? For example, would dissolving in tiles and then dissolving the entire shapefile be more efficient?

closed as too broad by PolyGeo Nov 13 '18 at 1:23

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • That it also my experience, dissolve is much faster in ArcGIS. You could try postgis, it is also very fast with a clever query. See the answer to Only union/dissolve intersecting features to save time? – BERA Nov 12 '18 at 21:06
  • I wonder if this workaround of rasterizing-polygonizing would be usable for your needs gis.stackexchange.com/questions/222976/…. – user30184 Nov 12 '18 at 21:13
  • @user30184 good catch, but the shapefiles I have already came from a raster in the first place, so I don't see any benefits of repeating the cycle. – Chouroud Nov 12 '18 at 21:15
  • @BERA postgis may be a solution. I will admit we were really hoping to find a Shapely solution since the rest of the backend is all done using geopandas. From your experience, are there any tricks that may speed up the geopandas dissolve? – Chouroud Nov 12 '18 at 21:20
  • I would suggest to try PostGIS as well blog.cleverelephant.ca/2009/01/…. Perhaps QGIS and GDAL does not implement the fast route even they do use GEOS. – user30184 Nov 12 '18 at 21:30