Ways to Speed Up Python Scripts Running As ArcGIS Tools [closed]

Question

This is a pretty general question. I just wondering what tips and tricks GIS programmers have used to speed up arcpy scripts that you import into the toolbox and run.

I work most everyday writing little scripts to help non-GIS users at my office process GIS data. I have found that ArcGIS 10.0 processing in general is slower than 9.3.1 and sometimes it gets even slower when running a python script.

I'm going to list a particular example of a script that takes over 24 hours to run. It's a loop that tabulates the area of a raster in a buffer for each shape in the buffer. The buffer has about 7000 shapes. I don't believe it should run this long. A

while x <= layerRecords:

    arcpy.SetProgressorLabel("Tabulating Row: " + str(x) + " of " + str(ELClayerRecords))
    arcpy.SelectLayerByAttribute_management(Buff,"NEW_SELECTION", "Recno = " + str(x))                                  # Selecting the record
    TabulateArea(Buff, "Recno", MatGRID, "VALUE", ScratchWS + "/tab" + str(z) +".dbf", nMatGRIDc)                          # Tabulate the area of the single row

    arcpy.AddMessage ("          - Row: " + str(x) + " completed")
    x = x + 1
    z = z + 1

Before anyone says it, I have run tabulate area on the entire buffer, but it produces errors if run on more then 1 record. It's a flawed tool, but I have to use it.

Anyways, if anyone has any ideas on how to optimize, or speed up this script, it would be most appreciated. Otherwise, di you have any speed up tricks for python, when used in ArcGIS?

Answer 1

General python optimization techniques can save you substantial amounts of time.

One really good technique for getting a lowdown of where the hold ups are in your script is using the built-in cProfile module:

from cProfile import run
run("code") # replace code with your code or function

Testing using a small data sample will allow you to pinpoint which function calls are taking the most time.

General pointers for faster python code:

• List comprehensions are generally faster than looping
• Generators produce one item at at time instead of producing the whole list at once
• Use xrange instead of range in python 2 (not necessary in 3)
• Sets can out preform lists when it comes to determining if an item is present in the set but are generally slower than lists when it comes to iterating over their contents Source
• Function calls can be costly to performance Source
• More tips and details check here Python Perfomance Tips and here 10 Python Optimization Tips and Issues

With regards to your script, I can't comment on the ArcPy aspects as i don't have Arc installed on this computer but you might want try using a for loop instead of a while loop see if that improves anything. Also x = x + 1 can be written as x+=1:

for record in layerRecords:
arcpy.SetProgressorLabel("Tabulating Row: " + str(x) + " of " + str(ELClayerRecords))
arcpy.SelectLayerByAttribute_management(Buff,"NEW_SELECTION", "Recno = " + str(x))                                  # Selecting the record
TabulateArea(Buff, "Recno", MatGRID, "VALUE", ScratchWS + "/tab" + str(z) +".dbf", nMatGRIDc)                          # Tabulate the area of the single row

arcpy.AddMessage ("          - Row: " + str(x) + " completed")
x+=1
y+=1

Answer 2

A couple potential suggestions to help speed up your process are:

1. Select Layer By Attribute can be in a Python-only script, without ever launching ArcGIS Desktop. You need to convert your "buff" reference from a file-based reference to an "ArcGIS layer" reference, which ArcGIS can process selection queries against. Use arcpy.MakeFeatureLayer_management("buff","buff_lyr") above your "while" loop, and then change your references below the while loop to use "buff_lyr".

2. Process as much of your GP operations using the in_memory workspace as possible... Use arcpy.CopyFeatures_management(shapefile, "in_memory\memFeatureClass") to move your source into memory. This only works well if you have enough RAM to read all of the feature class(es) that you need into memory. Beware, however, that there are some GP operations that cannot run using the in_memory workspace (Eg: the Project tool).

From ArcGIS 9.3 online help article "Intermediate data and the scratch workspace" (note, this language was removed from the 10.0 & 10.1 help):

NOTE: Only tables and feature classes (points, lines, polygons) can be written to the in_memory workspace. The in_memory workspace does not support extended geodatabase elements such as subtypes, domains, representations, topologies, geometric networks and network datasets. Only simple features and tables can be written.

From ArcGIS 10.1 online help article "Using in-memory workspace":

The following considerations must be made in deciding to write output to the in-memory workspace:

• Data written to the in-memory workspace is temporary and will be deleted when the application is closed.
• Tables, feature classes, and rasters can be written to the in-memory workspace.
• The in-memory workspace does not support extended geodatabase elements such as subtypes, domains, representations, topologies, geometric networks, and network datasets.
• Feature datasets or folders cannot be created in the in-memory workspace.

Answer 3

Make sure you are writing to internal drive on the computer. Reaching across the network when it is not necessary can really slow the processing. It can even be faster to copy the data as the first step in the process to keep the subsequent read-writes as quick as possible

Running the script completely outside of ArcMap can be much faster. If a Map isn't required during the processing, then don't use ArcMap.

Answer 4

This may not answer your question for running ArcPy tools inside ArcMap but when I need to do some meaty processing with geo-processing tools and Python I tend to run it outside the GIS system using the IDE PyScripter. I have found it runs faster. I have also employed a RAMDISK for small temporary output datasets (a bit like the in_memory workspace)

Well they are my top tips! :)

Answer 5

Try commenting out arcpy.SetProgressorLabel and see how much you speed up. I've found that any screen output, going back to DOS daze, drastically slows processing times. If you really need to see that output, trying showing it every Nth loop.

Answer 6

Make sure that you remove any import xxxx lines that aren't being used.

(ie. if you're not using any mathematical functions yet you have import Math, this will take some time from the script loading)

Although this will not have a great impact on single scripts which run (such as yours), it will effect any scripts that run frequently and repetitively.