I've recently tried using Shapely to dissolve and intersect large sets of 100'000+ geometries, some of them quite complex. This would take me at least one hour, at which point I would usually shut the process down.

I've been recommended PostGIS a few times as a super fast alternative, and I am wondering if it would indeed be faster? From my understanding both PostGIS and Shapely use GEOS, so would there really be such a speed difference between the two?

closed as primarily opinion-based by ahmadhanb, John Powell, nmtoken, xunilk, HDunn Dec 16 '18 at 16:37

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 4
    You should try and find out. Speed will depend on you precise scenario. But the benefits of PostGIS will be enhanced indexing (I don't know if you are using spatial indexes with Shapely), better data access (PostgreSQL will handle it efficiently). – HeikkiVesanto Nov 29 '18 at 14:08
  • 1
    @HeikkiVesanto Thanks for the answer. I agree that it's best to try out, but there is a cost to learning and setting up PostgreSQL and PostGIS. If it ends up saving me only a few minutes per dataset, I'm not really sure it's worth experimenting with it. – Chouroud Nov 29 '18 at 14:45
  • 1
    There is a cost to learning and setting up everything. I am a Pythonista and Postgres/Postgis lover, so no deliberate bias here, but I would argue that for larger datasets and more complex queries, the Postgres/Postgis environment is a better environment, plus you get all the RDBMS benefits thrown in. Learning of alternative ways of solving a problem is always a good idea. – John Powell Nov 30 '18 at 9:12
  • 1
    anecdotal: I recently had to intersect a set of 100 million polygons with a set of 33 million. My first pass at a Shapely solution was looking like it would have taken months to execute, and my PostGIS/Django solution takes about a day to run. – Teddy Ward Nov 30 '18 at 21:09
  • @TeddyWard great to hear. I'm very keen on making the leap to PostGIS. Any idea what makes it faster? Is it the chunk-based data access that makes all the difference? I'm coming from a programming background so I'm curious what's going on in the backend! – Chouroud Nov 30 '18 at 21:28