Is it possible to use Python to perform network analysis on only part of a network?

Question

I have a large network dataset and want to perform an analysis on it which would output the start/end point, the distance between them and whether they're first, second or third (etc.) nearest neighbours. There are approximately 30,000 nodes in the dataset, so I thought it might be possible to process 1,000 at a time (still comparing to all 30,000 nodes, but only doing the analysis on part of the dataset). I have an idea of how to do it (using a for loop, creating new files for each part of the analysis and putting it together) but I'm not certain as to how to create a for loop to use only part of the data and to go through and eventually finish analysis on all data.

Does anyone know if this is possible?

Edit: I've been having thoughts about the partitioning by rowID and was wondering then if I were to be making a service area analysis layer if then the layer would be impossible to create due to the partitioning or otherwise incorrect because of it?

Answer 1

I may have misunderstood what you want here but I have a couple of suggestions:

1. Split your data by doing a selection and then iterating the selection set. Here I would feed a variable into a SearchCursor and iterate the rows in the cursor with a for loop. So, you have a loop that sets yourt variable (by, say, increments of 100 and you select by FID > oldVar AND FID > newVar).
2. You say this is CPU intensive. If you have a PC with multiple cores, you could try farming this out with subprocesses in multiple threads. ArcGIS is not multithreaded (even in version 10 it is limited). You need to balance your demands though. Let's say you have a four-core PC. Without profiling this process, I would suggest leaving one core for the operating system and then spawning three subprocesses. I know it doesn't sound like a lot but that does equate to roughly 3-times as fast! Don't push it though because if you spawn too many threads you can end up slowing your machine down, not speeding it up. I've done this a lot in the past and my gut feeling for CPU-heavy processes in Arc is to go for your CPU count minus one. OR... if you are short of time and haven't used threading before, set three versions of your program going with a stepped increment in each so that process 1 is stepping through the first third, process two is doing the second third and so on. Yes! That is Heath Robinson, but you may need results faster than it would take you to learn multithreading even in Python.