I have always found such great analytical and programming expertise on StackExchange in the past that I hope you can help me.

Background: I have a project that involves spatially joining and interpolating traffic volume data across a road network on a continental scale. In order to handle the data processing, I have broken the data down by state and then into 10km processing tiles. States will run on parallel processes, which call a subprocess module to actually do the data processing in a 10km area at a time until the state level script is finished. The interpolation process starts with a road segment that has joined traffic data and searches connecting street segments with matching road name until it finds another segment with data. Then the code interpolates for segments in the middle.

The problem: When I run a single 10km tile or even ~20-30 tiles (for the same state), I get interpolation values that are expected given the algorithm I defined. When I start processing the entire state (thousands of tiles) with the very same code, I will get interpolation values for the same road that exceed the endpoint counts by orders of magnitude in either the positive or negative direction (and how could my traffic volume go negative, right?). Considering that the calculations will change for the same road depending on how much data is being processed, I feel that this is somehow more of a scaling or instability issue with Python, and not a bug in the code logic.

I was initially doing all the looping by referencing nested ArcGIS searchcursors (agh, experience backs up all the forum recommendations to NOT do that). So, I used an ArcGIS 10.0 searchcursor just once to read the street segment geometry and needed attributes into a Python dictionary for faster "lookups" during looping. The performance over ArcGIS searchcursors (even the new arcpy.da module) is amazing. I can now process the entire continent's worth of street data in two days. But now I'm wondering if Python dictionary tables can become instable if they get too large, or if the subprocess is not properly releasing memory objects before returning to the master script. There isn't any data passed from the subprocess back to the master; it's written to a permanent geodatabase table before closing. I wondered initially if the problem was running parallel processes, but I get the same unstable behavior for a single process on the state-level. I'm really not an expert Python programmer, though I've been effective in adapting other people's solutions to large data processing. This feels like one of those questions that a real developer could answer, as opposed to a code "pirate" like me.

I don't know that posting all my scripts will be helpful, as I think I'm looking for someone to point me in the right direction to research possible issues/solutions with python/subprocess instability, dynamic changing of values in dictionaries, etc. I think if I had a good starting place on issues that cause that kind of behavior, I could problem solve from there (hopefully).

  • 2
    If you can abstract out the ArcGIS-specific aspects of your question I would suggest posting it to StackOverflow. – blah238 Sep 27 '13 at 17:59
  • 1
    Could you describe how you recombine the individual results? That seems to be where it goes wrong. – blah238 Sep 27 '13 at 18:34
  • 1
    Can you load up your dictionaries using other than ArcPy and use them in skeleton code that emulates your overall procedure? Doing that could test your theory that large dictionaries become unstable. – PolyGeo Sep 28 '13 at 4:16