Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I'm attempting to clip a large polygon dataset (~5gb, thousands of features) by a much smaller polygon dataset (~40 features). Is there a best practice or most efficient route for performing this task?

The standard geoprocessor clip runs indefinitely on a dataset of this size. Would some form of spatial selection and export be more efficient?

EDIT: Some great answers below. I selected what I view to be the most thorough response, but each answer provides unique insight into the issue. Thanks!

share|improve this question
4  
How does the extent of the smaller dataset compare to that of the larger one? If the extents are very different, have you considered first clipping (or merely selecting via intersection) the large dataset with the extent of the smaller one to reduce the size of the problem? –  whuber Feb 1 '12 at 19:26
2  
Clipping very large polygon datasets in ArcGIS? Don't do it. After many years of trying this we end up using a database and get SQL the work, time is money. –  Mapperz Feb 1 '12 at 19:38
    
@whuber - +1 The extents are quite similar, although I've used your suggested method in the past. –  Radar Feb 1 '12 at 19:45
    
Too bad... I agree heartily with Mapperz. I also see @dmahr has posted a very nice reply that should help you sneak up on a solution or work-around, if one is possible. It provides a good strategy for dealing with any large dataset using any software. –  whuber Feb 1 '12 at 19:53
    
From this blog it seems to be quite problematic in ArcGIS. donmeltz.com/blog/index.php/2011/06/11/… –  Nicklas Avén Feb 1 '12 at 21:40
add comment

4 Answers

up vote 10 down vote accepted

As always when dealing with scalability problems, it's best to start small and simple and steadily work your way up to big and complex.

In the case of clip, it should be smart enough to deal with big datasets because it tiles them internally. But since it's not working, try running Clip with the input dataset (the data to be clipped) and the clip dataset (the data with which the clip is performed) with many, many fewer features. Like one clip feature, with only the area around it in the input dataset (use definition queries to shrink them). Make sure that it's running okay, and then steadily increase the scope of the geoprocessing operation until performance degrades.

A couple specific ideas:

  • Dissolve the clip features into a single, multi-part feature class.

  • Reduce the file size of the input features using Simplify Polygon. A 5GB vector dataset is enormous--even a shapefile of all 250,000 US Census block groups is only about 1GB.

  • Split the input features into parts. Theoretically the internal tiling routines within the geoprocessing tool should be doing this already, but you never know. There may be some 32-bit file size limitation issue where you can't have a shapefile bigger than 232 bytes = 4.29GB.

Some other, more general geoprocessing performance tips:

  • Make sure both datasets have the same coordinate system. If possible, it's faster to have both in a geographic coordinate system with no projection.

  • Make sure you are not running off of a network drive. Use the fastest local hard drive or, if possible, an SSD.

  • Load the clip dataset into memory.

  • Delete unnecessary attribute fields (and rejoin them later if needed).

  • Other geoprocessing performance tips.

share|improve this answer
4  
+1 Very nice. Might I suggest approaching the problem a little more quantitatively? Rather than just waiting for the onset of performance degradation, take the opportunity to time the operations. Even something as crude as watching the system clock can be useful. Plotting time against the size of the problem can indicate how it scales, give you a way to extrapolate the effort to the full problem, and even reveal places where the scaling changes (you hit a resource wall or the internal algorithm changes, for instance). It's easy information to get and can be very useful. –  whuber Feb 1 '12 at 19:56
    
I totally agree with Whuber on this. I always keep track of geoprocessing performance time, especially with timedelta objects in Python (when I'm scripting with ArcPy). Another tip is to use a more fully featured system monitor, like Process Explorer. This can give you much more information about performance and help you identify bottlenecks. –  dmahr Feb 1 '12 at 20:01
add comment

Or, you could give up on ArcGIS and try doing the clip in OGR. See an explanation here. I've found this works when nothing else does!

share|improve this answer
1  
ogr2ogr -clipsrc clipping_polygon.shp output.shp input.shp Damn, that is beautiful. –  R.K. Feb 2 '12 at 1:25
add comment

Some ArcGIS functions like Union and Intersect use adaptive subdivision processing as described in Tiled processing of large datasets. Unfortunately it appears as though Clip does not have a large geoprocessing tool.

It seems like you might be able to replicate this concept via a ModelBuilder or Python script that uses the Split tool (ArcInfo) to help process oversized datasets.

share|improve this answer
add comment

I think even select by location will work for you, first you select all the polygon which intersect with your smaller shapefile and then delete them. hopefully this might be quicker. lemme know if it works out for you. Thanks.

share|improve this answer
    
Not a bad idea in general--but did you read the comments to the original question? The preliminary query is not going to reduce the size of the problem much. –  whuber Feb 3 '12 at 20:16
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.