4

I'm trying to build a "large" network dataset from OSM data in ArcGIS. Doing it all at once is taking a "long time" and seems like there is a big opportunity for it to fail at some point which would cause me to have to start over. I would like to be able to ingest smaller dataset so that the processing would not take so long.

Is it possible to merge multiple smaller network datasets to a single dataset (that doesn't require rebuilding the entire network) that I can make routing calls to? Or maybe to do routing calls across multiple network datasets?

For reference the dataset I used was an OSM extract for all of California. It is 11GB of xml. It took me 4.5 days to turn that into feature classes. It took another 5.5 days to create the network dataset. The gdb is now 19.6GB. This was running on a m3.medium. I would eventually like to be able to route anywhere in the US.

  • 1
    What do you consider large? – Uffe Kousgaard Jul 30 '15 at 21:23
  • I downloaded the OSM extract for all of California. It is 11GB of xml. It took me 4.5 days to turn that into feature classes. It's still creating the network dataset but the gdb is now 18GB. I would eventually like to be able to route anywhere in the US. – Gabriel Littman Jul 30 '15 at 21:35
  • 1
    Please use the edit button beneath your question to revise it with clarifications such as this. Comments can be considered temporary, and are not always read by potential answerers. – PolyGeo Jul 31 '15 at 2:09
  • I tried the same California dataset and turned it into a routable SHP file in 20 minutes using our free osm2routeware converter (see www.routeware.dk). The SHP file is just 719 MB in total, so I am not sure what the 18 GB gdb file contains. But you could try to load the SHP file into network analyst rather than the OSM file directly. Doing it for all of the US isn't possible with a SHP file. It would get too big. – Uffe Kousgaard Jul 31 '15 at 6:59
  • You can also try and see Python Solving Large Network Problema from devSummit2018: proceedings.esri.com/library/userconf/devsummit18/papers/… – nicogis Sep 11 '18 at 14:03
6

As an answer to both Uffe Kousgaard comments about "what the 18GB file contains" compared to a routable shapefile, and a possible answer to this question:

  • You don't explicitly state it, but I guess you used the ArcGIS Editor for OpenStreetMap to convert your data. If not, I really recommend to have a look at it, as it contains a dedicated option to create routable networks of the converted data.
  • The Editor by default converts all data of the OSM file, meaning you end up not only with all highway line elements as necessary for building your network, but all other stuff like landuse and building polygons as well. The line Feature Class will contain other stuff as well, like waterways. This explains the 11-18GB data size compared to a lean routable highway network.
  • You could therefore consider to select features with a highway=x only, and export these to a dedicated Feature Class, preferably in a separate File Geodatabase as well. Please note (I have never created a routable network myself using the tools in the ArcGIS Editor for OpenStreetMap), that you may need to export other features as well, depending on what features are required by the Editor to build a routable network.
  • If you are just interested in routing for vehicles, discarding highway=footway, highway=bridleway, highway=cycleway, highway=pedestrian, highway=steps and highway=path should sharply reduce the size of the routable network as well.
  • I would strongly recommend to mildly generalize your features before creating a network. This will discard a lot of unnecessary detail / vertices / nodes that don't add anything to the routable network besides ballast. However(!), if you do this, you must use the options for maintaining topology during generalization. The Simplify Line tool of ArcGIS has an option for this. You must set:
    1. BEND_SIMPLIFY if you want to maintain the initial shapes as accurately as possible, POINT_REMOVE for speed.
    2. Set CHECK for the "error_checking_option"
    3. Set RESOLVE_ERRORS for the "error_resolving_option". Please note that this option can significantly increase generalization processing time, but since it is vital for maintaining the topology of the routable network, you must use it.
    4. Set an appropriate tolerance. You could start with something like 25 meter as a tolerance. 25 meters may sound like a lot when considering e.g. detailed small junctions or roundabouts, but please note that since you will select RESOLVE_ERRORS, ArcGIS will iteratively reduce this tolerance whenever connecting features would become topologically broken, until the generalized features are topologically correct while still somewhat generalized. So even setting values above 25 meters (e.g. 100 meter), should still retain basic topology of the network using the RESOLVE_ERRORS option. However, even setting a tolerance of 25 meters, is likely to reduce your Feature Class's size by maybe something like 90%.
  • Yes I'm using the ArcGISEditor for OpenStreetMap, I actually came across it only after I had found osm2nds which I believe ArcGISEditor uses under the hood. I'm novice at using it but I'll have to try what you are saying and see if I can limit the features it includes. – Gabriel Littman Aug 1 '15 at 0:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.