Arcpy scripts fail on big datasets - anyone has experienced similar behavior?

Question

Here a question that is not extremely specific but maybe someone has had similar experiences...

I have a few Arcpy scripts that work perfectly when used with small datasets but that silently fail when processing big datasets.

E.g. I am using Network Analyst tools to determine, all over the Swiss road network, the accessibility of various services (hospital, mall, bakery, school...). I basically create an OD matrix containing the distance between each populated place and the closest service. Apparently nothing rocket-science.

When my script is runnung against a small subset, it performs well and spits out a result. When run against the full road network it (usually) doesn't explicitely fail but all the output tables are just empty.

I experience a similar behavior when trying to perform a kind of custom filter on a point dataset: when using a few hundreds of points, no problem but when processing the full 4 Million points, the script runs without any warning but outputs an empty table.

I am aware that the question is quite vague but it's on purpose! As I seem to experience similar problems in unrelated situations, I would assume that there is a more global issue to address before trying to work on those specific scripts...

So, has anyone experienced similar behavior?

FYI: ArcGIS 10.1, Python 2.7, Windows Server 2008, 24GB RAM, Intel Xeon 2.67Ghz (2 processors)

Answer 1

I am going to suggest that you use a few of the profiling tools in Python to give you some insight as to what is going on with your script. You may have an undetected memory leak or you may be running out of memory as suggested.

TL;DR;

I would encourage you to seek visibility into how your script's performance and memory management or else you are just taking guessing. I have very little actual knowledge in this area as it has been a recent focus of mine, but I would be more than happy to assist.

In the Python standard library there is a cProfile. It is really easy to use and does not require you to modify your code. I would run it first on the smaller dataset to get a feel for what may be occuring.

Here is how you would run it from the command line:

$ python -m cProfile -o <stats file> <script to profile>.py

Once you have the stats file you can process it using the pstats module which is also included, here is a nice explanation how to use it. Another tool is objgraph and in the link there are some other good resources that the bottom. And here is another resource for finding memory leaks in Python.