Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

My question is: what might cause the ArcGIS 9.3 Hydrology Toolbox Snap Pour Point and Watershed tools to run significantly slower in a Python script versus manually running the tools in ArcCatalog (both runs using the exact same input data files and specification parameters)??

I'm running this script from the windows command line and an IDE (Komodo) so neither ArcMap or ArcCatalog are open during those runs; I'm using ArcGIS 9.3.1 on a Windows XP machine with 4 GB of RAM and plenty of processor power (a Dell Precision machine); and all input data are local; all data are in the same projection.

I have pre-processed Flow Accumulation, Flow Direction, and a Feature Class of water-quality sample sites to be used as pour points, and all datasets are in NAD83 UTM 17N projection. The Flow Accum and Flow Dir data files have spatial extents based on Hydrologic Unit Code (HUC) eight digit boundaries. The script iterates through each record in the sample site feature class, determines which eight digit HUC the point falls in, selects and exports the site record as a new feature class in a scratch geodatabase, and then handles the watershed processing using that exported point feature class.

I've stripped out comments and some other calls to a "processing log timestamp" function for reporting tool run times just to focus here on the question at hand and have posted the focus section of the script below. The biggest time sinks based on the processing log reporting are the Snap Pour Point tool and Watershed tool. We're talking on the order of 4+ minutes per site using a subset of all the sites for testing, and then 3-4 minutes for each site with the Watershed tool.

However, I can take the same single point feature class and the same flow accumuluation and flow direction rasters that the Python script is using, and run the Snap Pour Point and Watershed tools manually in ArcCatalog in approx 1 minute or less for each site (the resulting watersheds are the same as the watersheds the script produces). Example: one of the sites I manually ran the Snap Pour Point tool in 58 seconds via ArcCatalog, while the Python script executing that tool took 3 minutes 59 seconds for the tool to complete. I then ran the Watershed tool manually for the same site and it took 1 minute 1 second in ArcCatalog, while the Python script executing the Watershed tool took 4 minutes 8 seconds for the tool to complete. I'm using a 50 meter buffer for the Snap Pour Point tool in both the Python script and during my ArcCatalog manual run comparisons.

As you can see, CONSIDERABLY slower in the command line Python script. I've tried running this in an IDE (Komodo) and then via the Windows command line...same results with both. I've hit a wall with what I can figure out, so much appreciate any suggestions or insight.

thanks, Tom

try:
    if hucGdb in gdbFiles:
      outScratch = scratch + '\\' + delinSiteID + 'temp'
      flowAccum = filePath + '\\' + hucGdb + '\\' + 'FA_' + huc
      flowDir = filePath + '\\' + hucGdb + '\\' + 'FD_' + huc

      tempEnvironment = gp.extent
      gp.extent = flowAccum
      clause = '[DELIN_ID] = ' + "'" + delinSiteID + "'"
      gp.Select_analysis(delinSites, outScratch, clause)
      gp.extent = tempEnvironment

      outSnapRaster = snapPoint + '\\' + 'p' + delinSiteID

      gp.SnapPourPoint_sa(outScratch, flowAccum, outSnapRaster, tolerance)
      gp.extent = tempEnvironment

      outSnapFeature = snapPoint + '\\' + 'p' + delinSiteID + '_snap'
      gp.RasterToPoint_conversion(outSnapRaster, outSnapFeature, "VALUE")

      gp.AddField_management(outSnapFeature, "ORIGINAL_SITE_ID", "TEXT", "#", "#", "20")
      gp.CalculateField_management(outSnapFeature, "ORIGINAL_SITE_ID", '"' + realID + '"')

      gp.AddField_management(outSnapFeature, "DELIN_ID", "TEXT", "#", "#", "20") 
      gp.CalculateField_management(outSnapFeature, "DELIN_ID", '"' + delinSiteID + '"')

      outWsRaster = scratch + '\\' + delinSiteID
      tempEnvironment = gp.extent
      gp.extent = flowDir

      gp.Watershed_sa(flowDir, outSnapRaster, outWsRaster)
      gp.extent = tempEnvironment

      outWsFeature = basinOutput + '\\' + delinSiteID

      gp.RasterToPolygon_conversion(outWsRaster, outWsFeature, "SIMPLIFY")

      gp.AddField_management(outWsFeature, "DELIN_ID", "TEXT", "#", "#", "25") 
      gp.CalculateField_management(outWsFeature, "DELIN_ID", '"' + delinSiteID + '"')

      gp.AddField_management(outWsFeature, "ORIGINAL_SITE_ID", "TEXT", "#", "#", "25")
      gp.CalculateField_management(outWsFeature, "ORIGINAL_SITE_ID", '"' + realID + '"')

else:
  msg = """Site ID """ + realID + """ does not have a geodatabase of
  data, moving onto the next site"""
  print msg
  logFile.write(msg + '\n')

except:
print "ARCGISSCRIPTING ERROR: " + gp.GetMessages(2)
msg = "ARCGISSCRIPTING ERROR: " + gp.GetMessages(2)
logFile.write(msg + '\n')
print "ERROR: " + ErrorDesc.message
msg = "ERROR: " + ErrorDesc.message
logFile.write(msg + '\n')

EDIT

Just to see what happens, I set up a new ArcToolbox Toolbox and added my python script as a script that I could run. I have the script running based on argument input from a "configuration" text file that it reads in, so in this fashion I just double click on the script and click "OK" as there are no variables to set. I ran the script via ArcToolBox in ArcCatalog, not ArcMap.

The script ran in 7 minutes 32 seconds vs. 20+ minutes for my same three test sites (same file formats, everything), about one-third of the total time when running the script as a standalone.

Running the script in ArcCatalog definitely shaved off a bunch of time on the Snap Pour Point and Watershed tool runs (everything else was already reasonably fast so not worried about other tools/functions) based on my time stamp (tool started...tool finished) reporting I have built in.

SO....it seems that there is something going on (or perhaps in this case not going on) when you run a python script outside of ArcGIS (via command line or an IDE) that incurs considerable overhead on raster-calculation tools like Watershed and Snap Pour Point

I don't have any special settings (at least that I recognize/am aware of) in the ArcToolBox environments. As far as I know, these are the defaults (I've never messed with them - see attached screenshot). How would these (or a lack of these defaults) impact such tools - is that the issue? If so, can they be set in standalone python code??

I think this is getting closer.....much appreciate any suggestions

enter image description here

share|improve this question
    
Exact duplicate of... forums.arcgis.com/threads/… –  Dan Patterson Apr 11 '11 at 18:29
    
I will post a link in the unanswered question to the forum post that has an answer, or post the answer, etc. I've found there is overlap in terms of forum members between these two but there are some who dwell in only one since one is software-specific and the other is not –  turkishgold Apr 11 '11 at 19:38
    
Have you tried this process outside of a geodatabase? I don't use them, but I am wondering if there is any overhead running these types of processes outside of arcmap. It would be an interesting comparison on your part. –  Dan Patterson Apr 11 '11 at 21:10
    
I have not yet, but that seems like a strange drawback to scripting outside of arcmap or arccatalog in the language that they promote to use for scripting. I'm hoping there's some sort of gp parameter akin to "extent" or something else that can be set or tweaked in the python code that, outside of using the actual tools in arcmap or arccatalog, is more laborious resulting in the significant overhead for those two specific tools..... –  turkishgold Apr 11 '11 at 21:17
    
There is an extent option... webhelp.esri.com/arcgisdesktop/9.3/… is that what you are looking for? –  Dan Patterson Apr 11 '11 at 22:54

1 Answer 1

up vote 1 down vote accepted

So through discussions on another forum I was able to figure out why the standalone script was running so terribly slow. This is because I did not have the gp.ScratchWorkspace property set. Once I set that to mirror my ArcToolBox General Environment setting using the same directory, I was able to run the Python script from the command line in approx. 6 minutes 49 seconds, about 40 seconds quicker than the run I did using the script set up in ArcCatalog (using all the same input/output files and formats as I had been).

Someone had also suggested the gp.LoadSettings method to pull in the Environmental settings from an XML file, which I tried doing by saving out the Environment from ArcCatalog to XML and then pointing to that XML file with the latter gp method. It kept returning an error and saying the object could not be found, and I think this is a known bug as I found this separate discussion forum which alludes to this problem in the context of ArcObjects (but seems the same thing was happening with Python): http://forums.arcgis.com/threads/7204-igeoprocessor.savesettings-and-igeoprocessor.loadsettings-not-persisting-settings

So, this gp.ScratchWorkspace was exactly the "type" of setting I was hoping for that would solve the problem (a 2 second fix that gives you a "d'oh!" moment but let's you quickly forge ahead), something that on the surface seems not a big deal but in the end it more than tripled the total run time for the script when executed from the command line outside of ArcGIS.

thanks for the comments/discussion

Tom

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.