2

I'm trying to load huge .osm files into R in order to run advanced analysis. I did not expect the loading part to be complex, but it proved difficult.

The issue here is that the time it takes to parse the XML file (.osm is xml-like) and populate an object (here an osmar one) seems to be quadratically proportional to the file size.

require(osmar)
my_file = osmsource_file("filename")
osmar_object = get_osm("bla", source = my_file)

And in case you wonder how I found that it was quadratic, here is a representation of the analysis, with files between 0 and 500 Mo.

file size quadratically impacts loading time

At this point, my only chance to make it work seems to be the following : split the country in 100x100 small parts, and load them separately. Obviously, I don't really like the idea.

Why would it be so long ? How can I do so, fast ?

0
3

I decided to implement the split / combine approach successfully. The idea is to split a .osm file into n parts, load every single part and then combine them.

  1. The splitting is done with osmconvert, which provides an implementation with a bounding box. Here, I make sure every split has an overlap with its neighbors because that helps me ensure good merging performance.
  2. The loading part is done normally, with get_osm.
  3. The merging is done with a custom c() function, which code is available here : link

Thanks to this trick, was able to load countries in osmar in a few hours, which wasn't possible before (ex: France 7hours). The choice of the number of splits is the tricky part because it depends on the filesize. I tend to split in at least 50 parts, but sometimes you'll need far more.

I'm not happy with this answer though, because I still don't know why osmar ends-up taking quadratically more time loading large files. It does not make sense to me.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.