-1

I am trying to analyze several cities around the world (including major cities like London, Tokyo or Beijing) using different centrality measures with varying radiis.

The problem with this is that it takes several days for each city if I try to do it locally. So, my questions are:

(1) Is it better to move it to the cloud? If so, what is the best option in Amazon Web Services?

(2) Is there any other way to speed up the process locally?

0
2

Speeding up analysis locally

First question to ask is do you need betweenness centrality? This is the primary suck of compute time. Switch off betweenness and it's all much faster.

Also, do you really need radius n (global) or will something like 20km do the job? Large radii are a big time sink.

Another option is to sample only a subset of origins. Provided you sample enough (see below) the loss of precision won't be great - for all links, not just the origins sampled. There is a space syntax paper where they did this, but I don't have the reference to hand, sorry.

Ways to sample:

  1. If you use skipfraction=N in advanced config it will sample only 1 in N origins and the main part of the analysis will be N times faster. Note this is not random sampling so could create artifacts depending on how your data are ordered. (The feature is actually intended for parallelization - see below).

  2. Use skiporiginifzero=field_name - after creating field_name on your network and setting it to contain e.g. random samples from {0,1} with P(1)=0.01. Or whatever your requirements dictate. (If calculating closeness or network density within zones for example you could ensure there are N samples per zone, or whatever).

  3. One day I intend to add random sampling as a feature. Sorry it's not there right now.

What constitutes "enough" samples? I haven't done formal analysis of this yet, but generally speaking the larger the radius, the more origins you can discard. For each link in the network, you will need a handful of origins within radius of that link to be included in the sample. So for 400m radius you can't throw away many origins, but for 20km radius you could throw away loads. But smaller radii are faster anyway, remember, so it may well be worth doing multiple runs if you need multiple radii - small radius runs keeping most of the origins, large radius runs throwing most of them away.

Check your network for split links, fixing these with sDNA Prepare will reduce the network size.

Another way to speed things up that I have used on occasion is to remove minor roads (assuming they are tagged in your data). Be extremely careful though - I have come across errors in data sources usually considered reliable, where a short segment of major road is misclassified as minor, thus breaking connectivity when the minor road is removed.

Parallelization

sDNA will try to use all available shared memory cores on the machine. Ultimately there will be a limit to how much speed adding cores can buy you, as more time gets spent on concurrency overhead as each thread waits to increment output arrays. But I know of one customer who runs it just fine on 32 cores.

If you have a coder on board it shouldn't be hard to run sDNA across multiple machines, AWS, mapreduce, etc (I can't recommend any particular provider, but currently the licensing code is windows specific so you'll need windows installations on each machine). skipfraction and skipmod are intended for this approach; say you have N machines, run sDNA on each one with skipfraction=N;skipmod=i for i in 0..N-1.

Then you will need to combine the outputs from all machines. For betweenness you just add the corresponding results together for each link. For outputs which are means, e.g. mean angular distance, then adding will give the wrong answer. Possibly the best way is to add outputsums to advanced config then ignore "mean" type outputs, sum all the "sum" type measures and work out means right at the end by dividing by the corresponding sum by Links, Length or Weight whichever is appropriate to your analysis.

2
  • 1
    Nice answer Bob, I only understood about 3 words in 4 but you obviously know more than I do on this subject.. +1 from me. – Michael Stimson May 4 '18 at 12:12
  • Oh dear..! I hope that's just down to network analysis jargon and not a bad answer on my part. Feel free to ask for more info on anything in particular. – Sideshow Bob May 4 '18 at 12:56

Not the answer you're looking for? Browse other questions tagged or ask your own question.