3

Does anyone know what the fastest way to import OpenStreetMap data into an ArcGIS File Geodatabase (*.gdb) is? I'm using ArcGis 10.2.x and I see two options: OpenStreetMap Loader, which handles a *.bz2 input, or the file loader from the OpenStreetMap Editor toolbox, which handles a *.osm input.

I have to import about 100GB of OSM data into a Geodatabase. The data is all data with the 'highway' tag. Even splitting this up into smaller tiles with osmconvert and running these imports in parallel will take many days using the OpenStreetMap Editor file loader. In case it changes the answer, my computer can handle a lot (64GB RAM, nice processor and HDs etc).

  • Maybe you could use imposm or osm2pgsq and import OSM into PostGIS database and use PostGIS DB in your ArcGIS project. – dmitry.v.kiselev Aug 2 '15 at 15:58
  • Hmm good idea, I haven't tried importing into a PostgresSQL database yet because it makes the rest of the project more difficult and it didn't look much faster (wiki.openstreetmap.org/wiki/Osm2pgsql/benchmarks). Do you find that is? – Manselpotamus Aug 2 '15 at 18:41
  • I have not tried ArcGIS so I can't compare those two approaches. At least OSM + PostGIS is the most common way, especially for full planet coverage services. – dmitry.v.kiselev Aug 3 '15 at 5:26
2

You are not giving a whole lot of information here...

What version of the ArcGIS Editor for OpenStreetMap are you using? The latest version has seen some major improvements in terms of processing speed, and added much better multipolygon handling, meaning you get more accurate and better polygon output, close to what osm2pgsql achieves. Still, especially multipolygon creation, is a major technical hurdle and processing burden for any application working with OSM data.

What data do you need? The editor by default converts all data in an osm file, so if that data includes multipolygons as well, it may take a while... If you are just interested in a road network, it might be better to filter out the line data first using one the open source tools for that, and convert the remaining data to a File Geodatabase using the Load OSM File tool.

That said, 100GB is a lot, if you are not prepared to wait a couple of days for certain processes to finish, than you should probably not be handling this type of datasets at all... (unless you intend to do Big Data processing in a cluster, and have the facilities for that to shorten process time, ESRI has tools for putting data in a Big Data cluster).

Anyway, I have processed uncompressed XML osm of up to 25GB using the Editor's Load OSM File tool, and if I remember it well, for the latest release, it took around 3-4 days max on a Core i5 quadcore processor.

  • Where is the version number for ArcGIS Editor Toolbox? I don't see it on GitHub, in toolbox properties or on arcgis.com.. I edited the question to specify the data type I'm using, which is all road data as you guessed, I already used osmfilter and dropped everything I don't need. I don't have a Hadoop cluster either, thank you for that tip though. I've imported 25GB uncompressed XML files in ~30hrs 6 times now and I'm wondering if there's a faster way to do this, there's really no need to give career advice or make inferences about my patience when I ask for advice. – Manselpotamus Aug 2 '15 at 18:35
  • You can't see the exact version number, just make sure you have downloaded it lately, and are not using something from before the 4th of april 2015, which was the latest date ESRI has updated the toolbox (which you can see on the download page: arcgis.com/home/item.html?id=75716d933f1c40a784243198e0dc11a1. Thanks for the additional info, it clarifies more. There is one more optimization you might consider: drop any (route) line relations from the osm file if you just need the raw ways, as these will take processing time as well. – Marco_B Aug 2 '15 at 18:55
  • Lastly: a considerable amount of time is spent also on the node / point index creation. For a File Geodatabase, ArcMap will definitely be the bottle neck. Maybe a true enterprise geodatabase like PostgreSQL / Oracle or SQL Server, can do this faster using multiple processors (I haven't tried this in a true multiprocessor capable enterprise geodatabase, just SQL Server Express that can't take advantage of multiple processors) – Marco_B Aug 2 '15 at 19:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.