I have a large network dataset and want to perform an analysis on it which would output the start/end point, the distance between them and whether they're first, second or third (etc.) nearest neighbours. There are approximately 30,000 nodes in the dataset, so I thought it might be possible to process 1,000 at a time (still comparing to all 30,000 nodes, but only doing the analysis on part of the dataset). I have an idea of how to do it (using a for loop, creating new files for each part of the analysis and putting it together) but I'm not certain as to how to create a for loop to use only part of the data and to go through and eventually finish analysis on all data.

Does anyone know if this is possible?

Edit: I've been having thoughts about the partitioning by rowID and was wondering then if I were to be making a service area analysis layer if then the layer would be impossible to create due to the partitioning or otherwise incorrect because of it?

  • I thought of something... would this be possible with a while rather than a for loop?
    – Emily
    Apr 16 '12 at 15:58
  • 1
    You can exit a for loop using break after checking a condition such as x == 1000. However being unfamiliar with network analysis I can't say if this is the best approach.
    – blah238
    Apr 16 '12 at 18:37
  • 1
    You might be able to get some pointers from my partitioning technique in this answer: Port "Producing Building Shadows Avenue Code" to ArcGIS 10. This type of analysis may also lend itself to multiprocessing if you are finding speed is an important factor.
    – blah238
    Apr 16 '12 at 18:43
  • Thanks for sending that, the issue really is processor intenseness and I'm worried that it won't run all the way through a) quickly and b) without fail if I don't split it into part-processes.
    – Emily
    Apr 16 '12 at 18:53
  • wondering if you found any solution . did you tried QGIS network analysis ? Jun 22 '16 at 14:45

I may have misunderstood what you want here but I have a couple of suggestions:

  1. Split your data by doing a selection and then iterating the selection set. Here I would feed a variable into a SearchCursor and iterate the rows in the cursor with a for loop. So, you have a loop that sets yourt variable (by, say, increments of 100 and you select by FID > oldVar AND FID > newVar).
  2. You say this is CPU intensive. If you have a PC with multiple cores, you could try farming this out with subprocesses in multiple threads. ArcGIS is not multithreaded (even in version 10 it is limited). You need to balance your demands though. Let's say you have a four-core PC. Without profiling this process, I would suggest leaving one core for the operating system and then spawning three subprocesses. I know it doesn't sound like a lot but that does equate to roughly 3-times as fast! Don't push it though because if you spawn too many threads you can end up slowing your machine down, not speeding it up. I've done this a lot in the past and my gut feeling for CPU-heavy processes in Arc is to go for your CPU count minus one. OR... if you are short of time and haven't used threading before, set three versions of your program going with a stepped increment in each so that process 1 is stepping through the first third, process two is doing the second third and so on. Yes! That is Heath Robinson, but you may need results faster than it would take you to learn multithreading even in Python.
  • Thanks for your answer.. splitting the data would be impossible because I need to analyse the relationships between all nodes, and what I meant was to process just part by part by part but still examining the relationship for example between 1000 nodes and the whole and then save it, then the next 1000 and the whole then save it and so on. It's not a really urgent question, the data has to be cleaned and so on but I thought I'd get an early start on it.
    – Emily
    Apr 16 '12 at 20:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.