9

A python script was written about 18 months ago by a person who has now left. It produced the required outputs then. I've been asked to run it again but with different (finer resolution) data inputs. The input dataset has been split into 20 sub-sets of approx 2,700 data points each. However, the script crashes ("python.exe has stopped working") after approx 300 data points have been processed (range 295 to 306 and does NOT always fail on the same record).

As its old(ish), the script was written using arcgisscripting and not arcpy. Broadly it does the following using cursors:

  1. For a given point, calculate the cost distance (using gp.CostDistance_sa) with a cutoff of 60 minutes travel time.
  2. Calls gp.ExtractValuesToPoints_sa to extract all the individual values at each data point and outputs a feature class to a file geodatabase.
  3. Reads the feature class created in b) above and writes the values to a CSV file (omitting any points with "No Data" (value -9999)).

Repeats 1, 2 and 3 for all remaining data points in the input file.

Processing time is approx. 1 minute per data point on average. Here are some relevant technical specifications:

  • The PC has a quad core Intel i7-2720QM CPU running at 2.20GHz with 8GB RAM running Windows 7 (64 bit).
  • Python version is 2.6.6 (shell also states "[MSC v,1500 32 bit (Intel)] on win32).
  • ArcMap 10.0 (SP4) is also installed.

I've tried running it on a different PC (so far without crashing). Currently the job is running successfully (but more slowly) on an older PC and has reached 419 records without crashing. The relevant specifications for this machine are:

  • Intel Core 2 DUO E7500 processor running at 2.93GHz with 4 GB RAM and 64bit Windows 7.
  • Python version 2.5.1 (shell also states "[MSC v,1310 32 bit (Intel)] on win32).
  • ArcMap 9.3 is installed (no mention of any Service Packs).

Can someone offer some advice about why the script seems to work for a while then crash and how to resolve it?

The fact that a different PC appears (so far) to handle the script suggests something "environmental".

As an update, the PC running ARCGIS 9.3 is still successfully processing the data and has reached 1,300 data points processed (and still counting). A colleague also ran the data on their PC running ARCGIS 10.1 - it crashed after 267 records on two separate occasions. Although not conclusive, the common thread seems to be that Arc 9.3 will process the data but Arc 10.x will not.

Improve this question
7
  • 1
    ArcGIS 10.0 now uses the arcpy module (ArcGIS 9.x uses arcgisscripting module). You'll have to reconfigure your code to call arcpy as well as adjust the names of any geoprocessing tools, if you want it to work in AGS 10 environment.
    – dchaboya
    Commented Jan 10, 2013 at 16:20
  • 5
    No, thats not right - old scripts that worked in 9.3 will continue to work in 10 and 10.1. You do not need to modify gp to arcpy. You can even intermingle gp and arcpy throughout a script if you want to add new functionality, but not fully convert. ..... why this particular case is crashing above, I don't know. My suggestion is to break it down to sections and see exactly the last tool/function to happen before python bails
    – KHibma
    Commented Jan 10, 2013 at 16:59
  • KHibma, yeah I guesse that makes sense as it was partially working when running from AGS 10.
    – dchaboya
    Commented Jan 10, 2013 at 17:26
  • Have the data points changed? I assume you are using facilities within distance to the road network (travel time). There is no guarantee that the algorith for processing data points manages the points exactly the same every time the process is run. 300 or 306 or whatever could be conincidence. I've used NA for cost analysis of a network based on roads and locations in a python script and I'm wondering if you've tried a smaller subset. I would run far smaller groups of points on my workstation for 60 minute travel on my workstation. Travel time analysis will destroy processing power.
    – JLP Wisc.
    Commented Jan 10, 2013 at 20:49
  • 1
    Unfortunately we are also facing stability problems of arcpy GP (which is in fact just a COM wrapper, and as it looks like a buggy one). arcpy is the only site package I know, which can actually crash the python interpreter. CLJ suggested some workarounds down here in the answers (use 64 Bit GP, us GA cursors, etc...), but we already got respond from ESRI that these issues are bugs. Hopefully the next service pack brings improvement on this
    – Jürgen Zornig
    Commented Jun 28, 2013 at 21:14

3 Answers 3

Reset to default
1

If you run task manager and watch the python executable increase in memory and go over 1 gb before it dies, then you may benefit from upgrading to 10.1 64 bit geoprocessing.

For performance, if you are using cursors, you might benefit from the the new arcpy.da cursors. http://resources.arcgis.com/en/help/main/10.1/index.html#//018w00000008000000

I upgraded a project to use arcpy.da and it was a 2 magnitude improvement.

Improve this answer
1

This is an simply an arcpy bug. You can try to avoid using the steps that are causing the crash, but it generally happens under different tools when used to process through a long list of data. The only workaround I have found is to make my script save its progress along the way to disk so if you restart the process, it knows where to pickup from. If you then disable windows debugger message by altering the registry (see below), you can then just repeatedly execute the script in cmd.exe until it completes the entire batch without having to close the process manually every time in between.

I know this is an awful workaround, but it is quite uncommon to have a python library kill off the python interpreter. 

DWORD HKLM or HKCU\Software\Microsoft\Windows\Windows Error Reporting\DontShowUI = "1"
DWORD HKLM or HKCU\Software\Microsoft\Windows\Windows Error Reporting\Disabled = "1"
Improve this answer
0

Have you checked how the script handles cursors? My applications often hang when I forget to close them using explicit del row, cursor, sometimes only after some time.

If that doesn't help, I'd suggest using a smaller portion of code and/or data.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.