30

This is a pretty general question. I just wondering what tips and tricks GIS programmers have used to speed up arcpy scripts that you import into the toolbox and run.

I work most everyday writing little scripts to help non-GIS users at my office process GIS data. I have found that ArcGIS 10.0 processing in general is slower than 9.3.1 and sometimes it gets even slower when running a python script.

I'm going to list a particular example of a script that takes over 24 hours to run. It's a loop that tabulates the area of a raster in a buffer for each shape in the buffer. The buffer has about 7000 shapes. I don't believe it should run this long. A 

while x <= layerRecords:

    arcpy.SetProgressorLabel("Tabulating Row: " + str(x) + " of " + str(ELClayerRecords))
    arcpy.SelectLayerByAttribute_management(Buff,"NEW_SELECTION", "Recno = " + str(x))                                  # Selecting the record
    TabulateArea(Buff, "Recno", MatGRID, "VALUE", ScratchWS + "/tab" + str(z) +".dbf", nMatGRIDc)                          # Tabulate the area of the single row

    arcpy.AddMessage ("          - Row: " + str(x) + " completed")
    x = x + 1
    z = z + 1

Before anyone says it, I have run tabulate area on the entire buffer, but it produces errors if run on more then 1 record. It's a flawed tool, but I have to use it.

Anyways, if anyone has any ideas on how to optimize, or speed up this script, it would be most appreciated. Otherwise, di you have any speed up tricks for python, when used in ArcGIS?

|improve this question

closed as too broad by PolyGeo♦ Nov 20 '15 at 3:11

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

25

A couple potential suggestions to help speed up your process are:

  1. Select Layer By Attribute can be in a Python-only script, without ever launching ArcGIS Desktop. You need to convert your "buff" reference from a file-based reference to an "ArcGIS layer" reference, which ArcGIS can process selection queries against. Use arcpy.MakeFeatureLayer_management("buff","buff_lyr") above your "while" loop, and then change your references below the while loop to use "buff_lyr".

  2. Process as much of your GP operations using the in_memory workspace as possible... Use arcpy.CopyFeatures_management(shapefile, "in_memory\memFeatureClass") to move your source into memory. This only works well if you have enough RAM to read all of the feature class(es) that you need into memory. Beware, however, that there are some GP operations that cannot run using the in_memory workspace (Eg: the Project tool).

From ArcGIS 9.3 online help article "Intermediate data and the scratch workspace" (note, this language was removed from the 10.0 & 10.1 help):

NOTE: Only tables and feature classes (points, lines, polygons) can be written to the in_memory workspace. The in_memory workspace does not support extended geodatabase elements such as subtypes, domains, representations, topologies, geometric networks and network datasets. Only simple features and tables can be written.

From ArcGIS 10.1 online help article "Using in-memory workspace":

The following considerations must be made in deciding to write output to the in-memory workspace:

  • Data written to the in-memory workspace is temporary and will be deleted when the application is closed.
  • Tables, feature classes, and rasters can be written to the in-memory workspace.
  • The in-memory workspace does not support extended geodatabase elements such as subtypes, domains, representations, topologies, geometric networks, and network datasets.
  • Feature datasets or folders cannot be created in the in-memory workspace.
|improve this answer
  • 1
    That's fantastic! I've been looking for a way to use selections outside of ArcMap but, had been unsuccessful thus far. In terms of this problem, it actually pushed my time per row down to about 13 seconds from 20 seconds. But, I did a quick other work around and did the MakeFeatureLayer within the loop and it went down to 9 seconds. I did this by making a feature from each shape than tabulating from the feature layer. I'd still like to get it down further if possible, but already it's a much faster process! – Cody Brown Aug 20 '12 at 15:16
  • As mentioned in #2, use the CopyFeatures to make a copy of your source data in_memory, then create your feature_layer against the in_memory source. While the initial copy to memory may add a couple of seconds up-front, you may find that that processing of copyfeatures + tabulate_areas to have a faster total processing time than your current model. – RyanDalton Aug 20 '12 at 16:55
  • I tried that as well and it seems like that solution would make the loop process faster but, it doesn't. Creating the feature layer in the loop results in about 8-10 seconds per loop, while creating the feature layer before the loop results in 11 - 14 seconds per loop. I'm not too sure why since your solution sounds like it would process faster. I have 8GB of RAM, so I doubt that would be the problem. – Cody Brown Aug 20 '12 at 17:22
  • Also coping the features to in_memory before the loop and then still creating the feature layer in the loop results in a slightly faster performance. It pretty much stays a 8 seconds per row for each loop. Which will drop the total process time from 26 hours to 22. – Cody Brown Aug 20 '12 at 17:32
  • After adding your ideas my script dramatically improved. Thanks a ton for yours and everyone's help! – Cody Brown Aug 21 '12 at 17:27
28

General python optimization techniques can save you substantial amounts of time.

One really good technique for getting a lowdown of where the hold ups are in your script is using the built-in cProfile module:

from cProfile import run
run("code") # replace code with your code or function

Testing using a small data sample will allow you to pinpoint which function calls are taking the most time.

General pointers for faster python code:

  • List comprehensions are generally faster than looping
  • Generators produce one item at at time instead of producing the whole list at once
  • Use xrange instead of range in python 2 (not necessary in 3)
  • Sets can out preform lists when it comes to determining if an item is present in the set but are generally slower than lists when it comes to iterating over their contents Source
  • Function calls can be costly to performance Source
  • More tips and details check here Python Perfomance Tips and here 10 Python Optimization Tips and Issues

With regards to your script, I can't comment on the ArcPy aspects as i don't have Arc installed on this computer but you might want try using a for loop instead of a while loop see if that improves anything. Also x = x + 1 can be written as x+=1:

for record in layerRecords:
arcpy.SetProgressorLabel("Tabulating Row: " + str(x) + " of " + str(ELClayerRecords))
arcpy.SelectLayerByAttribute_management(Buff,"NEW_SELECTION", "Recno = " + str(x))                                  # Selecting the record
TabulateArea(Buff, "Recno", MatGRID, "VALUE", ScratchWS + "/tab" + str(z) +".dbf", nMatGRIDc)                          # Tabulate the area of the single row

arcpy.AddMessage ("          - Row: " + str(x) + " completed")
x+=1
y+=1
|improve this answer
  • 1
    I used the two links you left on your last bullet and was able to really help my script with a few quick fixes! – Cody Brown Aug 20 '12 at 15:21
  • If I could award two correct answers I would. While your answer really offered a lot of ideas on how to speed up python, @RyanDalton offered the idea's that had the most impact. Thanks a ton! – Cody Brown Aug 21 '12 at 17:25
13

Make sure you are writing to internal drive on the computer. Reaching across the network when it is not necessary can really slow the processing. It can even be faster to copy the data as the first step in the process to keep the subsequent read-writes as quick as possible

Running the script completely outside of ArcMap can be much faster. If a Map isn't required during the processing, then don't use ArcMap.

|improve this answer
  • I've found that running a script inside a model from ArcCatalog (by itself inside a Calculate Value dialog) will process faster than running the same script from the ArcPy window in ArcMap. That's purely an anecdotal observation though. – Cindy Jayakumar Aug 20 '12 at 5:09
  • 1
    I believe I do need a map for Tabulate to work properly, but I am going to try this. If it works outside of ArcMap, I bet it would speed up. Also I am already running off the local disk, that already doubled the speed of the script. – Cody Brown Aug 20 '12 at 12:55
  • Sadly the Select does not work outside of ArcMap and it is necessary because I need to do the tabulate shape by shape. – Cody Brown Aug 20 '12 at 13:49
  • 3
    @CodyBrown- You are incorrect about Select not working outside of an ArcMap session. See my response about using the MakeFeatureLayer tool. – RyanDalton Aug 20 '12 at 15:09
  • Ryan is right. When the select tool is used on its own it creates a table view of either your spatial data or table data. When using it in either ModelBuilde or in a script, you have to create a view, and in your case, creat it by using the MakeFeatureLayer tool. – dchaboya Aug 20 '12 at 19:24
6

This may not answer your question for running ArcPy tools inside ArcMap but when I need to do some meaty processing with geo-processing tools and Python I tend to run it outside the GIS system using the IDE PyScripter. I have found it runs faster. I have also employed a RAMDISK for small temporary output datasets (a bit like the in_memory workspace)

Well they are my top tips! :)

|improve this answer
  • 2
    To muddy this answer somewhat, when running scripts from Python IDEs many inject a traceback function to assist in watching variables and other various debugging assistance. This function can massively slow down scripts if it does too much as it gets called ALL THE TIME, and sometimes this is installed implicitly without user intervention. There was a particular pathological case I observed where a Python script run in ArcMap ran in 4 minutes, while the same script from Wing IDE took 3 hours. As soon as it was run from Python.exe without Wing, it went back to ~2-3 minute runtime territory. – Jason Scheirer Aug 19 '12 at 20:33
  • 1
    i was having headache to tun my scripts on ArMap, sometimes i cannot completely , till i turned to Pyscripter, it can slash the execution time compared to Arcmap , without using any optimization tip. – geogeek Oct 12 '12 at 11:23
  • @JasonScheirer did you find the tweak in Wing to turn this off? I'm sure there is one. – Curtis Price Jun 5 '13 at 17:40
5

Try commenting out arcpy.SetProgressorLabel and see how much you speed up. I've found that any screen output, going back to DOS daze, drastically slows processing times. If you really need to see that output, trying showing it every Nth loop.

|improve this answer
4

Make sure that you remove any import xxxx lines that aren't being used.

(ie. if you're not using any mathematical functions yet you have import Math, this will take some time from the script loading)

Although this will not have a great impact on single scripts which run (such as yours), it will effect any scripts that run frequently and repetitively.

|improve this answer
  • 7
    I doubt any standard Python module takes more than a thousandth of the time it takes the arcpy module to initialize. – blah238 Aug 18 '12 at 4:24
  • 1
    @blah238 import Math was probably a bad example. Some of the larger ArcPy libraries, however, do take a considerable amount of time to load up. – nagytech Aug 18 '12 at 4:54
  • 1
    this still shaves off only seconds (at most!), not hours – Mike T Aug 18 '12 at 5:38
  • 1
    @MikeToews For scripts that run frequently and repetatively, a few seconds adds up over the course of a few days / weeks etc. Although this dosen't solve the OP's main issue, he did ask for general tips. – nagytech Aug 18 '12 at 8:18

Not the answer you're looking for? Browse other questions tagged or ask your own question.