9

I have been observing unusual performance with a Python geoprocessing script. The (attached) script performs the following actions:

  1. Use a search cursor to look up UTM zone corresponding to polygon features
  2. Create spatial reference object based on search cursor results
  3. Convert .csv to feature layer and then to a point feature class

I've noticed markedly different processing times based on how the script is run:

  • 32-bit processing using IDLE = 203 seconds
  • 32-bit processing foreground script tool = 91 seconds
  • 64-bit processing background script tool = 206 seconds

Why would this script perform so differently given the above conditions? I certainly would not expect the 32-bit script tool run in the foreground to be 2X as fast as the other methods.


import arcpy, os, time

###IDLE Parameters
##fc = r'C:\path\to\polygon\fc\with\utm\zones\and\features'
##outws = r'C:\out\location'
##arcpy.env.workspace = r'C:\workspace'

####################
## Script tool parameters
fc = arcpy.GetParameterAsText(0)    # Feature class
outws = arcpy.GetParameterAsText(1) # Folder
arcpy.env.workspace = arcpy.GetParameterAsText(2)   # Workspace
####################

# Tables are .csv
tables = arcpy.ListTables()

start = time.clock()

# Look up which UTM zone .csv features are in
for t in tables:
    quad = t[7:17]
    print quad
    whereClause = """ "QUADID" LIKE '%s' """ % quad
    with arcpy.da.SearchCursor(fc, ("QUADID","ZONE"), whereClause) as cursor:
        for row in cursor:
            if row[0] == quad:
                utmZone = row[1]
                if utmZone == 10:
                    sr = arcpy.SpatialReference(26910)  # NAD_1983_UTM_Zone_10N
                elif utmZone == 11:
                    sr = arcpy.SpatialReference(26911)  # NAD_1983_UTM_Zone_11N
                elif utmZone == 12:
                    sr = arcpy.SpatialReference(26912)  # NAD_1983_UTM_Zone_12N
                elif utmZone == 13:
                    sr = arcpy.SpatialReference(26913)   # NAD_1983_UTM_Zone_13N
                else:
                    print "The UTM Zone is outside 10-13"
            else:
                pass

    # Convert .csv to feature class
    try:
        outLayer = "in_memory"
        # Now with the sr defined, create the XY Event Layer
        arcpy.MakeXYEventLayer_management(t, "x", "y", outLayer, sr, "z")
        arcpy.FeatureClassToFeatureClass_conversion(outLayer, outws, t[7:17])
        arcpy.Delete_management("in_memory")
        end = time.clock()
        print "In_memory method finished in %s seconds" % (end - start)

    except:
        # Print any error messages
        print arcpy.GetMessages(2)

print "Processing complete"
  • 1
    How long does it take to import arcpy just on it's own? Is there a formating error in the post. Should the try: be inside the for loop? – Nathan W Jan 25 '14 at 2:43
  • 2
    I think @NathanW's point about import arcpy is worth considering first because it would seem that time is only required by the IDLE and 64bit routes of your three tests, but adding nearly two minutes seems excessive. Try running a tool which does nothing more than time the import of ArcPy. – PolyGeo Jan 25 '14 at 2:59
  • 3
    I would be pretty safe to say that it's the import arcpy line. Last time I used arcpy it was slow to import from outside. ArcGIS would have that already imported in its internal Python so the import is already cached. – Nathan W Jan 25 '14 at 5:49
  • 3
    @Nathan and others are absolutely correct. Running a process via IDLE or the commandline takes a hit when you call 'import arcpy'. However, you can get a trade-off for very large processes where you get the time 'back' through improved performance. Running a background process also has a time hit as ArcGIS effective starts another ArcMap session. Lastly, you also have other variables you need to eliminate in your trial such as what are the differences in hardware between your 32 and 64 bit machines and what other processes were consuming resources during your trial etc? – MappaGnosis Jan 25 '14 at 8:55
  • 2
    +1 @JayLaura. Could go further and profile. [General python doc][docs.python.org/2/library/profile.html] and [stackexchange posting][stackoverflow.com/questions/582336/…. – Roland Jan 27 '14 at 17:15
2

@Aaron: reposting my earlier comment as answer based on his advice:

Could go further and profile. [General python doc] and [stackexchange posting].

Definitely interested in hearing what he finds.

6

I have a theory.

I'm thinking the issue may be the validation of your output or your input. Before a GP tool runs, arcpy validates parameters, for example, whether the output feature class exists already.

Within ArcMap, the workspace (folder) contents are all cached and the validation can be done against catalog's "view" of the workspace -- in memory -- fast. This can cause confusion if datasets are added using a non-ArcGIS tool, requiring an arcpy.RefreshCatalog() to be run to synch up the catalog view with the state of the workspace (folder).

If your folder is very large, and you are running outside of ArcGIS, arcpy may be having to generate a folder listing each time to validate your FeatureClassToFeatureClass output. If there are many items in the folder, this could really get slow.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.