I have a large set of line data (>140,000 features). Is there a processing advantage, in either time required or (more importantly) memory used:

  • to running Dissolve on the data before running Buffer?
  • to running Dissolve on the inputs to two Identity operations?

I would generally just wait until all the geoprocessing is finished and then do one Dissolve at the end. However, I'm debugging somebody else's very old script, and I am unclear on whether he was repeatedly dissolving everything for a reason (back in Arc 9.3), or just didn't think about the alternatives. (The same script repeatedly projects data between geoprocessing tools, so the logic is already questionable.)

  • I don't have any hard data to back this up, but from my personal experience: always buffer before a dissolve if possible, because buffering such a complex line feature takes a freaking eternity. – nmpeterson Aug 27 '14 at 14:34

If memory use is your prime concern, then lots of little (low vertex count) features is probably going to be more to your liking than a few very large (high vertex count) features. But you may find that "too many features" may eventually overwhelm even "too many vertices" for processing speed.

If you think about how the algorithms must be structured to process all features against all features between two feature classes, you're working with multiply-nested loops (for features in FC1 and FC2, and for the vertices in Feature1 and Feature2). In operations like drawing, the number of draw requests is of often greater concern than the vertices in each request, but with theme-on-theme operations, the key algorithms are likely to be based on the counts of vertices in each F1/F2 pair, with a "big O notation" of "O(N*M)" (the time to complete the operation is related to the factor of the number of vertices involved), which, for large features in both datasets, is close enough to O(N^2) to make you worry about the job ever completing.

I've had success by overlaying massive features (like Russia, Canada, US, Australia, Brazil, Norway) with a 5 degree grid (fishnet) to reduce feature complexity for intermediate processing. I've seen point-in-polygon operations on a vertex-restricted 1:15m COUNTRIES layer run 100-1000 times faster than the original table (with only a 20x feature count increase). You do need to be careful in your processing logic to handle one-to-many and many-to-many relationships correctly though, especially in cases where a false boundary exists.

There's also a "diminishing returns" aspect to the savings of working with smaller features -- I settled on a 5-degree grid by testing performance of intersecting with 90, 45, 30, 20, 15, 10, 5, 3, 2 and 1-degree grids, which showed an alarming increase in processing time as the number of total features ballooned.

There are times where fewer features with more vertices are more efficient, so it is probably worth the effort to do some testing on order of operation with real data (not simplified test subsets) before committing to one approach over the other (balancing RAM utilization with run time).

NOTE: I re-ran the gridding exercise with modern hardware, and got optimal performance with a 30-degree overlay, so that increases the risk of too-small features, and increases the importance of evaluation with production data.

| improve this answer | |

A Dissolve operation will usually reduce the number of features, arcs and nodes within a layer, particularly for layers with significant lengths of shared boundaries. Since the time spent during a Buffering operation is highly dependent on the number of nodes, pre-processing with Dissolve may significantly reduce the running time (and memory requirements). Whether or not it is worthwhile in your case will depend on the extent to which you will be able to reduce the number of nodes (dependent on your data layer) and the efficiency of the Dissolve operation compared with the Buffering. In my experience, using the Java Topology Suite, a Dissolve operation can be quite fast compared to Buffering, although the performance of Dissolve varies significantly by library. The other consideration is that Dissolve is strongly affected by topological errors. If you layer contains errors, you will need to perform vector cleaning prior to the Dissolve operation, which will add to the workflow run time.

| improve this answer | |
  • 2
    I'm not so sure about the "memory requirements" part. Larger features require more storage. Buffering very complex features is more difficult (and RAM-intensive) than buffering simple features. It's more likely that there's a "sweet spot" between "too many features" and "too many vertices per feature" than making a blanket assertion that dissolving first will always improve buffer performance. – Vince Aug 27 '14 at 14:46
  • @Vince, I'll agree that the effect of Dissolving is much more effective at reducing the run time rather than the memory, but ultimately if a group of features has fewer features with fewer total nodes, it will require less memory to represent. – WhiteboxDev Aug 27 '14 at 14:58
  • It will reduce total memory, but not memory per feature. Performing geoprocessing operations on massive, complex features takes longer than on simpler features -- and not just in a linear manner, in my experience. – nmpeterson Aug 27 '14 at 15:06
  • @Vince, Also Dissolve will, depending on the layer, result in fewer larger features. The actual geographical size of a feature has nothing to do with its memory requirements. It's entirely down to its complexity, which is a function of the number of nodes used to represent it. I'll agree with you about the sweet-spot balance though. – WhiteboxDev Aug 27 '14 at 15:08
  • @nmpeterson, Yes, it's true that this is on a per layer and not per feature basis. But we buffer layers generally and not individual features. You sure are right about the non-linearity of performance in geospatial processing though! It seems that that is always the case for us! – WhiteboxDev Aug 27 '14 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.