Reduce processing time in Erase function using Arcpy in IDLE

Question

I am processing 18,000 feature classes (saved in a file GDB), each with ~500 records (polygon features) in ocean areas, and I am running a script to remove area overlapping with the coast, using the Erase function. The script is running successfully, but literally taking days to complete. Some files take seconds to process (I'm assuming this is due to the fact, which I've verified, that they are not overlapping with coastline) while some feature classes take over an hour to process.

I am working locally on my machine, running the script in IDLE, with ArcGIS 10.1 (all licenses) on a 32-bit Windows 7, 3.33GHz Intel DuoCore, 4GB RAM computer.

Are there any steps I can take to speed up processing time?

# Import modules
import arcpy
import sys
import os
from arcpy import env

arcpy.env.overwriteOutput = True
print arcpy.env.overwriteOutput

#Erase coastline from selected buffers.
workspace = "C:\\DIR\\points_buff\\"
arcpy.env.workspace = workspace
erase_output = workspace+"erased.gdb\\"
erasedirectorypath = "C:\\DIR\\points_buff\\selectedbuff.gdb"
fileList = []

for dirpath, dirnames, erases in arcpy.da.Walk(erasedirectorypath,
                                              datatype="FeatureClass",
                                              type="Polygon"):
    for erase in erases:
        fileList.append(os.path.join(dirpath, erase))
        filenameparts = erase.split('_')
        if int(filenameparts[1]) > int(1967):
        
            try:
        #Set variables for Erase tool
                in_features = erasedirectorypath+"\\"+erase
                coastline_feat = "C:\\DIR\\Basic_NE.gdb\\Coastline_Eastern_US"
                out_erase_feat = erase_output+erase[:-4]+"_er"
                arcpy.Erase_analysis(in_features, coastline_feat, out_erase_feat, '')
                print arcpy.GetMessages()
                print "Coastline has been erased from " + erase + " and saved to " + erase_output

            except:
                print arcpy.GetMessages()

Answer 1

Although it is difficult to troubleshoot your script without seeing the spatial data, a few tricks and ideas may help your workflow. To begin, subset your dataset and experiment with different approaches. Incorporate a time clock into your script to get a real sense of how fast or slow operations are. Try wrapping your commands with a time.clock. For example:

import time
StartTime = time.clock()
arcpy.Erase_analysis(in_features, coastline_feat, out_erase_feat, '')
EndTime = time.clock()
print "Erase finished in %s hours" % ((EndTime - StartTime)/3600)
time.sleep(2.5)

• Try utilizing the power of in_memory workspace. Generally, operations performed in_memory are faster than those that are written to disk. Use in_memory for any intermediate operations.

• Rather than using the Erase command, use the following set of commands to perform a similar workflow:

Make Feature Layer (Data Management)

Select Layer by Location (Data Management)

Delete Features (Data Management)

Copy Features (Data Management)

• Are your polygons overly complicated? Dissolve (Data Management) may help.
• Consider a raster-based approach, perhaps using a mask (if feasible).