How to import OpenStreetMap data into GraphX?

Question

How to import OpenStreetMap data into a Spark GraphX RDD?

Would like to see feasibility of doing drive-time analysis in our own Spark cluster.

Thought to use osm2pgsql or a similar project, but not sure if preserves data required for routing.

Looked at OSM2Routing, but it's not clear what config file required e.g. for whole U.S. map?

I'm new to both OSM and GraphX.

Update. GraphX needs two tables. 1st - Vertex Table (id of a node, properties); 2nd - Edge Table (SrcId, DstID, properties). https://spark.apache.org/docs/latest/graphx-programming-guide.html#example-property-graph

Answer 1

There are a couple of things coming to my mind here:

• First, and foremost, why you feel the need to use an unproven HPC routing solution to do something for which the OSM community already developed proven and tested solutions? Routing on OSM data has been implemented as several Open Source projects (See http://wiki.openstreetmap.org/wiki/Routing. Even ESRI has a routing (or routing network generation) capability build into their Open Source ArcGIS Editor for OpenStreetMap. Each of these run on ordinary hardware and ordinary configurations as far as I can tell, not requiring extensive clusters. Even the main OpenStreetMap website now features routing, disproving the entire need for something as complex as an HPC solution (and now I am even forgetting 10 year old commercial car routing computers that already could do the job). It sounds to me a bit like trying to catch a mouse with a shotgun...
• Extending on this: most OSM users don't have access to something as fancy as a HPC cluster. This essentially means that you are probably "on your own" with this question, and probably means you will need to dig deep and work hard to solve this, if you really intend to do this as some kind of research project. Your current question slightly raises doubt to me if you actually did your "homework" yet... You can't start a project like this if you are "New"... Start reading everything you can about OSM and GraphX!

• One thing that - possibly - provides a solution for getting OSM data in an HPC cluster (I am ignoring the in-memory versus disk-based here, as I am not really sure it is relevant), is the ESRI route, because I know ESRI also has been working on a solution to put GIS data in an HPC cluster and to extend Hadoop with OGC-based spatial functions. So one possible route might be something like:

  • 1) Import OSM data into a File Geodatabase, or Oracle / SQL Server Enterprise Geodatabase, using the Load OSM File tool part of the ArcGIS Editor for OpenStreetMap. An *.osm file for the entire US can be downloaded from Geofabrik.

  • (- 1b) Possibly use the Create OSM Network Dataset tool of this toolbox to create a routable network).

  • 2) Export data to JSON in the Hadoop cluster using the ESRI Geoprocessing tools for Hadoop toolset
  • 3) Possibly make use of the ESRI Geometry API for Java, developed for use on Hadoop, to do more fancy stuff on your Hadoop stored data, like building a routable network

I have done none of this ever myself, so bear with me if it is somewhat speculative, but I know the tools are there...

Answer 2

Checkout this tool that converts OSM PBFs to Parquet files:

https://github.com/adrianulbona/osm-parquetizer

.. and the following blogpost containing a demo where Spark is used to reconstruct Way geometries:

http://adrianulbona.github.io/2016/12/18/osm-parquetizer.html

Answer 3

I realized I had omitted another good link for you. There is an interesting blog by one of the ESRI developers involved in the HPC stuff (Mansour Raad). I hadn't visited it for a while, but it turns out he actually wrote an article about invoking a Spark job from within ArcGIS for Desktop using ArcGIS Geoprocessing tools. There is code samples as well. I don't know if you have any experience with ArcGIS, and especially Python as being the scripting language of the ESRI Geoprocessing framework, but it does show some of the feasibility of the suggested work flow of my first post in relation to Spark.

See here for Mansour Raad's blog: http://thunderheadxpler.blogspot.nl/2014/01/apache-spark-spatial-functions-and.html

And another remark: be aware that loading the > 140 GB *.osm uncompressed XML file of the entire US using the Load OSM File tool, may take up to two weeks! I recently imported a 12 GB uncompressed file of Poland, and loaded that in an ESRI File Geodatabase. If I remember it well, the process took about 1-2 days... There is unfortunately quite a lot of processing needed to convert the data, especially when OSM multipolygon relations are involved. The tool does a pretty good job at it, but it takes time.

Files on Geofabrik are in zipped format, compression ratio is about 1:12.

You may consider to filter the *.osm files to cut back on the workload of importing. There are tools to get specific data from them, in your case maybe only the highways, e.g. OSMOSIS, but I have no experience with these: http://wiki.openstreetmap.org/wiki/Osmosis

Lastly, prepare yourself for the fact (but I guess you may already know this), that things like address information in OSM, isn't that well structured. There are many variations.