Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I am processing 18,000 feature classes (saved in a file GDB), each with ~500 records (polygon features) in ocean areas, and I am running a script to remove area overlapping with the coast, using the Erase function. The script is running successfully, but literally taking days to complete. Some files take seconds to process (I'm assuming this is due to the fact, which I've verified, that they are not overlapping with coastline) while some feature classes take over an hour to process.

I am working locally on my machine, running the script in IDLE, with ArcGIS 10.1 (all licenses) on a 32-bit Windows 7, 3.33GHz Intel DuoCore, 4GB RAM computer.

I'm learning the ropes with Python and would really appreciate tips on my script if there are steps I can take to speed up processing time. Thank you!

# Import modules
import arcpy
import sys
import os
from arcpy import env

arcpy.env.overwriteOutput = True
print arcpy.env.overwriteOutput

#Erase coastline from selected buffers.
workspace = "C:\\DIR\\points_buff\\"
arcpy.env.workspace = workspace
erase_output = workspace+"erased.gdb\\"
erasedirectorypath = "C:\\DIR\\points_buff\\selectedbuff.gdb"
fileList = []

for dirpath, dirnames, erases in arcpy.da.Walk(erasedirectorypath,
                                              datatype="FeatureClass",
                                              type="Polygon"):
    for erase in erases:
        fileList.append(os.path.join(dirpath, erase))
        filenameparts = erase.split('_')
        if int(filenameparts[1]) > int(1967):

            try:
        #Set variables for Erase tool
                in_features = erasedirectorypath+"\\"+erase
                coastline_feat = "C:\\DIR\\Basic_NE.gdb\\Coastline_Eastern_US"
                out_erase_feat = erase_output+erase[:-4]+"_er"
                arcpy.Erase_analysis(in_features, coastline_feat, out_erase_feat, '')
                print arcpy.GetMessages()
                print "Coastline has been erased from " + erase + " and saved to " + erase_output

            except:
                print arcpy.GetMessages()
share|improve this question
add comment

1 Answer 1

up vote 6 down vote accepted

Although it is difficult to troubleshoot your script without seeing the spatial data, a few tricks and ideas may help your workflow. To begin, subset your dataset and experiment with different approaches. Incorporate a time clock into your script to get a real sense of how fast or slow operations are. Try wrapping your commands with a time.clock. For example:

import time
StartTime = time.clock()
arcpy.Erase_analysis(in_features, coastline_feat, out_erase_feat, '')
EndTime = time.clock()
print "Erase finished in %s hours" % ((EndTime - StartTime)/3600)
time.sleep(2.5)
  • Try utilizing the power of in_memory workspace. Generally, operations performed in_memory are faster than those that are written to disk. Use in_memory for any intermediate operations.

  • Rather than using the Erase command, use the following set of commands to perform a similar workflow:

Make Feature Layer (Data Management)

Select Layer by Location (Data Management)

Delete Features (Data Management)

Copy Features (Data Management)

share|improve this answer
2  
Calculating time can be invaluable for comparing methods, especially if you are trying many different approaches on a small subset of data. –  Aaron Apr 24 '13 at 14:22
2  
Yes, keep in mind that you can easily delete in_memory files within your script by using Delete (Data Management). –  Aaron Apr 24 '13 at 14:29
4  
How many features are in the coastline file, and what is the typical number of vertices in a feature? How large of an area does it cover? How large of an area do your feature classes typically cover? If you have very large and complicated features in your coastline file, that might be the source of the slow down. There are a few methods to use to adjust that if it is the problem. –  blord-castillo Apr 24 '13 at 15:09
2  
I would actually go the opposite route. First dissolve the features, then break the big polygon up into several separate feature classes of one polygon each. Try running just one of these subsetted single feature coastline feature classes against one of your feature classes that took a while (and intersected). See if that gives you any improvement. I think your intersection envelopes might simply be too big, taking away any of the performance tweaks already built into overlay analyses. –  blord-castillo Apr 24 '13 at 18:47
4  
Another workflow to test. Run Minimum Bounding Geometry on your target feature class using Group Option ALL and geometry type ENVELOPE. Perform an erase on the resulting envelope feature class using your enormous coastline. Use that output from that erase to do a Clip on your original target feature class. This simplifies the complex step, the erase, and substitutes it with a less costly Clip against a smaller feature. –  blord-castillo Apr 24 '13 at 18:56
show 12 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.