We have many datasets, each containing about 150'000 polygons. We would like to dissolve each dataset down to about 2500 polygons, based on a given attribute.

We've successfully done this using the ArcGIS Dissolve tool, but we would like to move towards an open-source solution. The problem is that every open source solution we have tried so far is incredibly slow. It takes ArcGIS about one to two minutes to dissolve one dataset. With other methods, it takes at least one hour, at which point I stop it since my guess is it has crashed completely due to memory issues.

These are the methods we have tried so far:

  1. QGIS Dissolve tool (NOT the Saga or GRASS tools). I use QGIS 2.18, would QGIS3 make a big difference?
  2. GeoPandas/Shapely.
  3. ogr2ogr (as described here http://darrencope.com/2015/01/23/dissolve-shapefiles-using-ogr/)
  4. We would gladly try other tools such as GRASS, but we require this to be able to run on a machine for which we do not have admin rights. This means no installations or sudo commands.

Are there are there other tools or methods we can try? Are there some "tricks" to try out? For example, would dissolving in tiles and then dissolving the entire shapefile be more efficient?

  • That it also my experience, dissolve is much faster in ArcGIS. You could try postgis, it is also very fast with a clever query. See the answer to Only union/dissolve intersecting features to save time?
    – BERA
    Nov 12, 2018 at 21:06
  • I wonder if this workaround of rasterizing-polygonizing would be usable for your needs gis.stackexchange.com/questions/222976/….
    – user30184
    Nov 12, 2018 at 21:13
  • @user30184 good catch, but the shapefiles I have already came from a raster in the first place, so I don't see any benefits of repeating the cycle.
    – Chouroud
    Nov 12, 2018 at 21:15
  • @BERA postgis may be a solution. I will admit we were really hoping to find a Shapely solution since the rest of the backend is all done using geopandas. From your experience, are there any tricks that may speed up the geopandas dissolve?
    – Chouroud
    Nov 12, 2018 at 21:20
  • I would suggest to try PostGIS as well blog.cleverelephant.ca/2009/01/…. Perhaps QGIS and GDAL does not implement the fast route even they do use GEOS.
    – user30184
    Nov 12, 2018 at 21:30