4

How to import OpenStreetMap data into a Spark GraphX RDD?

Would like to see feasibility of doing drive-time analysis in our own Spark cluster.

Thought to use osm2pgsql or a similar project, but not sure if preserves data required for routing.

Looked at OSM2Routing, but it's not clear what config file required e.g. for whole U.S. map?

I'm new to both OSM and GraphX.

Update. GraphX needs two tables. 1st - Vertex Table (id of a node, properties); 2nd - Edge Table (SrcId, DstID, properties). https://spark.apache.org/docs/latest/graphx-programming-guide.html#example-property-graph

Improve this question
3
  • crosspost: stackoverflow.com/questions/30010269/… – scai May 6 '15 at 7:39
  • Which modes of transport are you looking into? Just car? Or also walking, biking, ...? – underdark♦ May 10 '15 at 11:53
  • Car primarilly for U.S. Walking would be nice to have but not mandatory. Not sure if we'll need biking. Thanks! – Tagar May 10 '15 at 22:25
3
+100

There are a couple of things coming to my mind here:

  • First, and foremost, why you feel the need to use an unproven HPC routing solution to do something for which the OSM community already developed proven and tested solutions? Routing on OSM data has been implemented as several Open Source projects (See http://wiki.openstreetmap.org/wiki/Routing. Even ESRI has a routing (or routing network generation) capability build into their Open Source ArcGIS Editor for OpenStreetMap. Each of these run on ordinary hardware and ordinary configurations as far as I can tell, not requiring extensive clusters. Even the main OpenStreetMap website now features routing, disproving the entire need for something as complex as an HPC solution (and now I am even forgetting 10 year old commercial car routing computers that already could do the job). It sounds to me a bit like trying to catch a mouse with a shotgun...
  • Extending on this: most OSM users don't have access to something as fancy as a HPC cluster. This essentially means that you are probably "on your own" with this question, and probably means you will need to dig deep and work hard to solve this, if you really intend to do this as some kind of research project. Your current question slightly raises doubt to me if you actually did your "homework" yet... You can't start a project like this if you are "New"... Start reading everything you can about OSM and GraphX!

  • One thing that - possibly - provides a solution for getting OSM data in an HPC cluster (I am ignoring the in-memory versus disk-based here, as I am not really sure it is relevant), is the ESRI route, because I know ESRI also has been working on a solution to put GIS data in an HPC cluster and to extend Hadoop with OGC-based spatial functions. So one possible route might be something like:

    • 1) Import OSM data into a File Geodatabase, or Oracle / SQL Server Enterprise Geodatabase, using the Load OSM File tool part of the ArcGIS Editor for OpenStreetMap. An *.osm file for the entire US can be downloaded from Geofabrik.

    • (- 1b) Possibly use the Create OSM Network Dataset tool of this toolbox to create a routable network).

    • 2) Export data to JSON in the Hadoop cluster using the ESRI Geoprocessing tools for Hadoop toolset
    • 3) Possibly make use of the ESRI Geometry API for Java, developed for use on Hadoop, to do more fancy stuff on your Hadoop stored data, like building a routable network

I have done none of this ever myself, so bear with me if it is somewhat speculative, but I know the tools are there...

Improve this answer
2
  • 2
    Thanks a lot for detailed respone. We already have a commercial routing solution and it does not scale to millions of addresses well. It takes weeks to process whole data set. On the other hand we already have Hadoop cluster and we do have Spark services running there, we have commerical support for that. So using GraphX feels like a natural way to go for us; hope that partially answers of "why". Also, it is easy to scale-out performance in Hadoop cluster just by adding more nodes. It's not so easy to scale performance with other solutions. Thanks - I will dig deeper into the links you posted. – Tagar May 10 '15 at 22:19
  • Yes, this explains more about the background, and why you want to do this. It may be an interesting test case for this kind of setup (and probably simply a necessity for you). – Marco_B May 11 '15 at 18:28
3

Checkout this tool that converts OSM PBFs to Parquet files:

https://github.com/adrianulbona/osm-parquetizer

.. and the following blogpost containing a demo where Spark is used to reconstruct Way geometries:

http://adrianulbona.github.io/2016/12/18/osm-parquetizer.html

Improve this answer
1
  • That's great! Thanks for sharing this. A next logical step for us would be to use that data for drive-time analsysis.. Have you tried this? – Tagar Apr 20 '17 at 15:35
2

I realized I had omitted another good link for you. There is an interesting blog by one of the ESRI developers involved in the HPC stuff (Mansour Raad). I hadn't visited it for a while, but it turns out he actually wrote an article about invoking a Spark job from within ArcGIS for Desktop using ArcGIS Geoprocessing tools. There is code samples as well. I don't know if you have any experience with ArcGIS, and especially Python as being the scripting language of the ESRI Geoprocessing framework, but it does show some of the feasibility of the suggested work flow of my first post in relation to Spark.

See here for Mansour Raad's blog: http://thunderheadxpler.blogspot.nl/2014/01/apache-spark-spatial-functions-and.html

And another remark: be aware that loading the > 140 GB *.osm uncompressed XML file of the entire US using the Load OSM File tool, may take up to two weeks! I recently imported a 12 GB uncompressed file of Poland, and loaded that in an ESRI File Geodatabase. If I remember it well, the process took about 1-2 days... There is unfortunately quite a lot of processing needed to convert the data, especially when OSM multipolygon relations are involved. The tool does a pretty good job at it, but it takes time.

Files on Geofabrik are in zipped format, compression ratio is about 1:12.

You may consider to filter the *.osm files to cut back on the workload of importing. There are tools to get specific data from them, in your case maybe only the highways, e.g. OSMOSIS, but I have no experience with these: http://wiki.openstreetmap.org/wiki/Osmosis

Lastly, prepare yourself for the fact (but I guess you may already know this), that things like address information in OSM, isn't that well structured. There are many variations.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.