Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

A python script was written about 18 months ago by a person who has now left. It produced the required outputs then. I've been asked to run it again but with different (finer resolution) data inputs. The input dataset has been split into 20 sub-sets of approx 2,700 data points each. However, the script crashes ("python.exe has stopped working") after approx 300 data points have been processed (range 295 to 306 and does NOT always fail on the same record).

As its old(ish), the script was written using arcgisscripting and not arcpy. Broadly it does the following using cursors: a) For a given point, calculate the cost distance (using gp.CostDistance_sa) with a cutoff of 60 minutes travel time.

b)Calls gp.ExtractValuesToPoints_sa to extract all the individual values at each data point and outputs a feature class to a file geodatabase.

c) Reads the feature class created in b) above and writes the values to a CSV file (omitting any points with "No Data" (value -9999)).

Repeats a), b) and c) for all remaiming data points in the input file.

Processing time is approx. 1 minute per data point on average. The PC has a quad core Intel i7-2720QM CPU running at 2.20GHz with 8GB RAM running Windows 7 (64 bit).

Python version is 2.6.6 (shell also states "[MSC v,1500 32 bit (Intel)] on win32).

Not sure if relevant but ARCMAP 10.0 (SP4) is also installed.

I've tried running it on a different PC (so far without crashing). Currently the job is running successfully (but more slowly) on an older PC and has reached 419 records without crashing. This has an Intel Core 2 DUO E7500 processor running at 2.93GHz with 4 GB RAM and 64bit Windows 7.

Python version 2.5.1 (shell also states "[MSC v,1310 32 bit (Intel)] on win32).

ARCMAP 9.3 is installed (no mention of any Service Packs).

Please can someone offer some advice about why the script seems to work for a while then crash and how to resolve it? The fact that a different PC appears (so far) to handle the script suggests something "environmental".

Thanks in anticipation.


Thanks to everyone for your suggestions. I'll try these. As an update, the PC running ARCGIS 9.3 is still successfully processing the data and has reached 1,300 data points processed (and still counting). A colleague also ran the data on their PC running ARCGIS 10.1 - it crashed after 267 records on two seperate occasions. Although not conclusive, the common thread seems to be that ARC 9.3 will process the data but ARC 10.x will not.

Any further contributions gratefully received.

share|improve this question
1  
ArcGIS 10.0 now uses the arcpy module (ArcGIS 9.x uses arcgisscripting module). You'll have to reconfigure your code to call arcpy as well as adjust the names of any geoprocessing tools, if you want it to work in AGS 10 environment. –  dchaboya Jan 10 '13 at 16:20
5  
No, thats not right - old scripts that worked in 9.3 will continue to work in 10 and 10.1. You do not need to modify gp to arcpy. You can even intermingle gp and arcpy throughout a script if you want to add new functionality, but not fully convert. ..... why this particular case is crashing above, I don't know. My suggestion is to break it down to sections and see exactly the last tool/function to happen before python bails –  KHibma Jan 10 '13 at 16:59
    
KHibma, yeah I guesse that makes sense as it was partially working when running from AGS 10. –  dchaboya Jan 10 '13 at 17:26
    
Have the data points changed? I assume you are using facilities within distance to the road network (travel time). There is no guarantee that the algorith for processing data points manages the points exactly the same every time the process is run. 300 or 306 or whatever could be conincidence. I've used NA for cost analysis of a network based on roads and locations in a python script and I'm wondering if you've tried a smaller subset. I would run far smaller groups of points on my workstation for 60 minute travel on my workstation. Travel time analysis will destroy processing power. –  JLP Wisc. Jan 10 '13 at 20:49
1  
Unfortunately we are also facing stability problems of arcpy GP (which is in fact just a COM wrapper, and as it looks like a buggy one). arcpy is the only site package I know, which can actually crash the python interpreter. CLJ suggested some workarounds down here in the answers (use 64 Bit GP, us GA cursors, etc...), but we already got respond from ESRI that these issues are bugs. Hopefully the next service pack brings improvement on this –  Jürgen Zornig Jun 28 '13 at 21:14
show 2 more comments

3 Answers

If you run task manager and watch the python executable increase in memory and go over 1 gb before it dies, then you may benefit from upgrading to 10.1 64 bit geoprocessing.

For performance, if you are using cursors, you might benefit from the the new arcpy.da cursors. http://resources.arcgis.com/en/help/main/10.1/index.html#//018w00000008000000

I upgraded a project to use arcpy.da and it was a 2 magnitude improvement.

share|improve this answer
add comment

Have you checked how the script handles cursors? My applications often hang when I forget to close them using explicit del row, cursor, sometimes only after some time.

If that doesn't help, I'd suggest using a smaller portion of code and/or data.

share|improve this answer
add comment

If not already, I would use a proper Python IDE.

We use PyCharm Professional by JetBrains .

Coincidentally, today while testing some of our old scripts that we are currently updating, I encountered a similar problem where the python interpreter would crash rather than throw an error. I was able to determine the exact part of the code and set a break point to determine that one of my intermediate datasets was missing a projection which was causing the failure.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.