Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

Here a question that is not extremely specific but maybe someone has had similar experiences...

I have a few Arcpy scripts that work perfectly when used with small datasets but that silently fail when processing big datasets.

E.g. I am using Network Analyst tools to determine, all over the Swiss road network, the accessibility of various services (hospital, mall, bakery, school...). I basically create an OD matrix containing the distance between each populated place and the closest service. Apparently nothing rocket-science.

When my script is runnung against a small subset, it performs well and spits out a result. When run against the full road network it (usually) doesn't explicitely fail but all the output tables are just empty.

I experience a similar behavior when trying to perform a kind of custom filter on a point dataset: when using a few hundreds of points, no problem but when processing the full 4 Million points, the script runs without any warning but outputs an empty table.

I am aware that the question is quite vague but it's on purpose! As I seem to experience similar problems in unrelated situations, I would assume that there is a more global issue to address before trying to work on those specific scripts...

So, has anyone experienced similar behavior?

FYI: ArcGIS 10.1, Python 2.7, Windows Server 2008, 24GB RAM, Intel Xeon 2.67Ghz (2 processors)

share|improve this question
1  
What type of failure? e.g. is there an MemoryError? I have 32-bit arcpy fail all the time when doing numpy operations on float rasters due to the memory limitations. –  blord-castillo Mar 4 at 13:16
1  
This happened to me all the time until I started using 64-bit background geoprocessing. You should make sure you are running your scripts utilizing the full power of your 64-bit system: resources.arcgis.com/en/help/main/10.1/index.html#//… –  Aaron Mar 4 at 13:39
2  
There is a new Quality Improvement Patch available for AGD 10.1 SP1: support.esri.com/en/downloads/patches-servicepacks/view/…. I haven't tried it as I work with 10.2.1, but it can't hurt to install it –  GISGe Mar 4 at 13:40
1  
I will have the quality patch installed but I'm not sure where I can find the Background-Geoprocessing (64bits) patch for 10.1? It's not listed here: bit.ly/1czaIS5 Has it disappeared? –  Stéphane Henriod Mar 4 at 14:48
2  
the background geoprocesssing installer can be found at support.esri.com/en/downloads/patches-servicepacks/view/… under the SP1 setup files –  Craig Mar 4 at 16:34

1 Answer 1

I am going to suggest that you use a few of the profiling tools in Python to give you some insight as to what is going on with your script. You may have an undetected memory leak or you may be running out of memory as suggested.

TL;DR;

I would encourage you to seek visibility into how your script's performance and memory management or else you are just taking guessing. I have very little actual knowledge in this area as it has been a recent focus of mine, but I would be more than happy to assist.

In the Python standard library there is a cProfile. It is really easy to use and does not require you to modify your code. I would run it first on the smaller dataset to get a feel for what may be occuring.

Here is how you would run it from the command line:

$ python -m cProfile -o <stats file> <script to profile>.py

Once you have the stats file you can process it using the pstats module which is also included, here is a nice explanation how to use it. Another tool is objgraph and in the link there are some other good resources that the bottom. And here is another resource for finding memory leaks in Python.

share|improve this answer
    
I'm having a look at it but haven't really understood yet how to take advantage of the outputed profile file. I will come back to you if I have more specific questions –  Stéphane Henriod Apr 9 at 13:30
    
Here you go. stefaanlippens.net/… –  Jamie Apr 9 at 13:31

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.