I have huge line and polygon shapefiles that I want to edit with some geoprocessing tasks (clip, buffer, erase,...). Altough my computer is not that slow, I am annoyed of waiting up to 15 hours for one of these tasks. Some people say that importing the shapefiles in a File Geodatabase will help. Are there any further hints? Is there any difference whether I perform such ArcGIS for Desktop tasks through the ArcCatalog or ArcMap applications?
-
How big are we talking here? Also maybe post your computer specs?– GISHumanSep 20, 2013 at 19:57
-
3Are you running Background Geoprocessing (64-bit)? It speeds things up considerably.– SeaJunkSep 20, 2013 at 19:58
-
The shapefiles are between 1.5 and 3.0 GB large. I am working with 10Gb of RAM, i7 CPU and SSD. The hint with the Background Geoprocessing sounds promising!– GideonSep 20, 2013 at 20:16
-
3Gb? A shapefile should not exceed 2Gb! Limitations are discussed here. You don't tell us anything about your data other than it's lines and polygons. 15 hours is along time and suggests to me its more to do with your data than arcmap. Are they over-lapping within a featureclass? This would negate any performance provided by spatial indexing. Please edit your question and describe your data a bit more.– HornbyddSep 21, 2013 at 15:38
5 Answers
ESRI has a blog post - aptly titled Dicing Godzillas - about processing feature classes with large amounts of vertices. It discusses the Dice tool, which splits up a shapefile's features into smaller features with less vertices.
You won't find much of a difference (if any) running in ArcCatalog vs. ArcMap. However, you might find some improvement by running it entirely outside of either by using Python. If your data would benefit from being split up, you could use Python for multiprocessing, which is discussed here.
Still another option (that can be leveraged via scripting or ArcMap) would be to use the in_memory workspace. This doesn't always work for really large files as you'll get memory overflow errors, but it might be worth a shot if you will be processing this dataset multiple times.
Depending on your data, you can try combining all of the above to see what works best for your situation and data.
Other than that, running it overnight is always an option. Good luck!
Edit @SeaJunk's comment: If you are running ArcMap 10.1 SP1 or above, definitely take a look at Background Geoprocessing. Great tip!
Any time you are looking at a heavy process consider scripting it out with python.
By scripting you do not have to worry about ArcMap or Catalog having to load all of the other overhead (such as the UI) that you do not need while running your process. If you have multiple steps, the script can launch right in to those with out waiting for you to click the button (you don't want to be there just to click the button at 1 am to get the next step going). Finally, by scripting it you make it easier to rerun the entire process in the future for any reason.
Yes, these last two can also be accomplished with a model, but you have to deal with the extra overhead of ArcGIS Desktop, you also lose out on better error handling and the flexibility that Python has to offer.
As for the file format, I recommend you get your data in to one of the geodatabase formats. The geodatabases seem to be less prone to file corruption than shape files. Up to 1GB you can use either the Personal Geodatabase (.mdb) or File Geodatabase (.gdb). Bigger than that the .mdb geodatabase starts to get flaky due to the design limitations of MS Access.
Another thing that will help significantly is storing your shapefiles (and your output files) on the computer's SSD, though it sounds from your comment like you're already doing that. Converting your (huge) shapefiles to file geodatabase feature classes will definitely help too, but not nearly as much as splitting your shapefiles into smaller chunks and processing them separately as Paul suggests.
-
Whilst all the chunks would have to be processed after another, wouldn't that add up to the same amount of time?– GideonSep 20, 2013 at 20:24
-
2No. Coordinate processing complexity is O(N*M), where N and M are the number of vertices in the compared shapes. If you cut shape vertex count, even by adding more shapes, you significantly reduce processing time.– VinceSep 20, 2013 at 21:44
Select a subset of your FIDs, like 20,000 at a time, do the geoprocessing, and then merge the results back together. I believe the geoprocessing tools are getting wound in the spatial tiling algorithm.
I am not sure if this will help you or not, but I have realized that if you have a very big polyline shapefile and you want to for example reshape it, it would be much more faster if you split it into smaller parts. To do this you may use a sort of paneling or a mesh polygon to split the polylines. This is going to reduce the process time considerably and in the end you have the option of merging them together.