Unusual results for geoprocessing speed tests

Question

I have been observing unusual performance with a Python geoprocessing script. The (attached) script performs the following actions:

1. Use a search cursor to look up UTM zone corresponding to polygon features
2. Create spatial reference object based on search cursor results
3. Convert .csv to feature layer and then to a point feature class

I've noticed markedly different processing times based on how the script is run:

• 32-bit processing using IDLE = 203 seconds
• 32-bit processing foreground script tool = 91 seconds
• 64-bit processing background script tool = 206 seconds

Why would this script perform so differently given the above conditions? I certainly would not expect the 32-bit script tool run in the foreground to be 2X as fast as the other methods.

---

import arcpy, os, time

###IDLE Parameters
##fc = r'C:\path\to\polygon\fc\with\utm\zones\and\features'
##outws = r'C:\out\location'
##arcpy.env.workspace = r'C:\workspace'

####################
## Script tool parameters
fc = arcpy.GetParameterAsText(0)    # Feature class
outws = arcpy.GetParameterAsText(1) # Folder
arcpy.env.workspace = arcpy.GetParameterAsText(2)   # Workspace
####################

# Tables are .csv
tables = arcpy.ListTables()

start = time.clock()

# Look up which UTM zone .csv features are in
for t in tables:
    quad = t[7:17]
    print quad
    whereClause = """ "QUADID" LIKE '%s' """ % quad
    with arcpy.da.SearchCursor(fc, ("QUADID","ZONE"), whereClause) as cursor:
        for row in cursor:
            if row[0] == quad:
                utmZone = row[1]
                if utmZone == 10:
                    sr = arcpy.SpatialReference(26910)  # NAD_1983_UTM_Zone_10N
                elif utmZone == 11:
                    sr = arcpy.SpatialReference(26911)  # NAD_1983_UTM_Zone_11N
                elif utmZone == 12:
                    sr = arcpy.SpatialReference(26912)  # NAD_1983_UTM_Zone_12N
                elif utmZone == 13:
                    sr = arcpy.SpatialReference(26913)   # NAD_1983_UTM_Zone_13N
                else:
                    print "The UTM Zone is outside 10-13"
            else:
                pass

    # Convert .csv to feature class
    try:
        outLayer = "in_memory"
        # Now with the sr defined, create the XY Event Layer
        arcpy.MakeXYEventLayer_management(t, "x", "y", outLayer, sr, "z")
        arcpy.FeatureClassToFeatureClass_conversion(outLayer, outws, t[7:17])
        arcpy.Delete_management("in_memory")
        end = time.clock()
        print "In_memory method finished in %s seconds" % (end - start)

    except:
        # Print any error messages
        print arcpy.GetMessages(2)

print "Processing complete"

Answer 1

I have a theory.

I'm thinking the issue may be the validation of your output or your input. Before a GP tool runs, arcpy validates parameters, for example, whether the output feature class exists already.

Within ArcMap, the workspace (folder) contents are all cached and the validation can be done against catalog's "view" of the workspace -- in memory -- fast. This can cause confusion if datasets are added using a non-ArcGIS tool, requiring an arcpy.RefreshCatalog() to be run to synch up the catalog view with the state of the workspace (folder).

If your folder is very large, and you are running outside of ArcGIS, arcpy may be having to generate a folder listing each time to validate your FeatureClassToFeatureClass output. If there are many items in the folder, this could really get slow.

Answer 2

@Aaron: reposting my earlier comment as answer based on his advice:

Could go further and profile. [General python doc] and [stackexchange posting].

Definitely interested in hearing what he finds.