I hope this question is appropriate here and I apologize if it is more of a discussion question than a specific technical problem.

I have to buffer 150,000 polygons for a real estate project. While it's not impossibly slow, it got me wondering about 'the fastest way to geoprocess'. What packages, tips, or tricks could be used to make this faster? Does anyone have an idea how arcmap, arcpy, shapely, postgres compare in performance time?

EDIT: After asking around, I heard 'arcpy is slow!' a lot.

  • 8
    Let the database do the work, ArcGIS is quite slow for large amounts of geo-processing. See ST_Buffer (PostGIS) postgis.refractions.net/documentation/manual-2.0/ST_Buffer.html – Mapperz Mar 5 '13 at 18:06
  • 16
    Think of ways to avoid buffering. IMHO, people overuse it because it's a conceptually simple, direct way to perform some kinds of fixed-distance-based analyses. In many cases buffering can be avoided altogether by structuring the analyses appropriately and exploiting other available methods of proximity-based calculation. The disadvantage is that buffering each polygon produces on average around several dozen more vertices surrounding each vertex, which in your case could produce billions of new vertices: that's worth avoiding. – whuber Mar 5 '13 at 18:14
  • 2
    Perhaps you could use this as a "learning experience" for your supervisor? Digging deeper into a simple request often reveals the "real reason" behind the request, and often you can not only demonstrate faster turnaround time, but also more accurate analysis. – RyanKDalton Mar 5 '13 at 19:11
  • 1
    While this blog (which references this article) does not answer faster alternatives for buffering, it may provide you some ideas on speed of geoprocessing operations for many software. It could at least give you some direction as to where you may want to start. – RyanKDalton Mar 5 '13 at 19:15
  • 1
    If you just need the cartography--which is a perfectly valid reason to create buffers--it at least opens up the prospect of computing the buffers with a raster analysis rather than a vector one. With so many polygons, the raster calculations ought to be much faster than the vector calculations, even at extremely high resolutions. Moreover, once this possibility is entertained, cartographic improvements become available. – whuber Mar 5 '13 at 19:57

You could also look at using the OGR library, it has python bindings so you could create an ArcGIS buffer tool pretty easily.

I find ArcGIS/ArcPy slow in general, so try and avoid it as much as possible.

I would also endorse using PostGIS, it eats this kind of processing for breakfast and isn't difficult to setup

Not the answer you're looking for? Browse other questions tagged or ask your own question.