Importing road graph from OSM file

Question

What I want to do:

I want to get road graph (ways and their relations) from OSM file to write my own routing program.

What I tried to do:

I tried to use OSMtoRoadGraph, but it returns me a UnicodeDecodeError

(base) C:\Windows\system32>python 
C:\Users\Horseman.mini\Desktop\OsmToRoadGraph-master\run.py -f 
C:\Users\Horseman.mini\Desktop\OsmToRoadGraph-master\data\siberian-fed-district-latest.osm_01_01.osm -n c
starting convert_osm_to_roadgraph
selected network type: car
accepted highway tags: {'pedestrian': {'path', 'secondary_link', 'footway', 'secondary', 'tertiary_link', 'pedestrian', 'track', 'primary', 'tertiary', 'primary_link', 'unclassified', 'road', 'steps', 'service', 'living_street', 'residential'}, 'bicycle': {'secondary_link', 'secondary', 'tertiary_link', 'cycleway', 'track', 'primary', 'tertiary', 'primary_link', 'unclassified', 'road', 'path', 'service', 'living_street', 'residential'}, 'car': {'secondary_link', 'motorway', 'motorway_link', 'secondary', 'trunk', 'tertiary_link', 'trunk_link', 'primary', 'tertiary', 'primary_link', 'unclassified', 'service', 'living_street', 'residential'}}
opening file: C:\Users\Horseman.mini\Desktop\OsmToRoadGraph-master\data\siberian-fed-district-latest.osm_01_01.osm
starting read_file
starting _read_ways
0%..Traceback (most recent call last):
File "C:\Users\Horseman.mini\Desktop\OsmToRoadGraph-master\run.py", line 65, in <module>
convert_osm_to_roadgraph(filename, network_type, options)
File "C:\Users\Horseman.mini\Desktop\OsmToRoadGraph-master\utils\timer.py", line 11, in wrapper
result = function(*args, **kwargs)
File "C:\Users\Horseman.mini\Desktop\OsmToRoadGraph-master\run.py", line 25, in convert_osm_to_roadgraph
nodes, ways = osm.read_osm.read_file(filename, configuration)
File "C:\Users\Horseman.mini\Desktop\OsmToRoadGraph-master\utils\timer.py", line 11, in wrapper
result = function(*args, **kwargs)
File "C:\Users\Horseman.mini\Desktop\OsmToRoadGraph-master\osm\read_osm.py", line 11, in read_file
ways, found_node_ids = _read_ways(PercentageFile(osm_filename), parserHelper)
File "C:\Users\Horseman.mini\Desktop\OsmToRoadGraph-master\utils\timer.py", line 11, in wrapper
result = function(*args, **kwargs)
File "C:\Users\Horseman.mini\Desktop\OsmToRoadGraph-master\osm\read_osm.py", line 23, in _read_ways
parser.parse(osm_file)
File "C:\ProgramData\Anaconda3\lib\xml\sax\expatreader.py", line 111, in parse
xmlreader.IncrementalParser.parse(self, source)
File "C:\ProgramData\Anaconda3\lib\xml\sax\xmlreader.py", line 123, in parse
buffer = file.read(self._bufsize)
File "C:\Users\Horseman.mini\Desktop\OsmToRoadGraph-master\osm\xml_handler.py", line 25, in read
data = self.f.read(size)
File "C:\ProgramData\Anaconda3\lib\encodings\cp1251.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x98 in position 993: character maps to <undefined>

I think the problem is that OSM uses utf-8 format, but I see in the error this line at the very end "C:\ProgramData\Anaconda3\lib\encodings\cp1251.py", line 23, in decode so unicode error is predictable. If no ways to solve unicode problem or any ways to fix this problem using this soft - I haven't found any other ways to import roads graph from OSM.

Answer 1

I'm the maintainer of this project and would like to help you out.

UPDATE I fixed the issue in version 0.42.1

I tried to reproduce the problem locally with data I downloaded from geofabrik.de (I guessed you did the same. I got siberian-fed-district-latest.osm). However, on my Mac/Linux the reported issue doesn't appear. You can also open an issue in the Github repo if you rather want to discuss this there.

It would be great if you could somehow provide a sample of the file that produces the problem. I also have a potential way to fix it, but can't really test it, since I can't reproduce the problem. But if you want. You could do the following:

• go to osm/ and open the file xml_handler.py
• replace l. 18 with self.f = open(filename, encoding="utf-8")
• try to run it again (and please tell me if it worked or not)

Lastly, I have a re-implementation (that is not yet finished) in Rust which not have this issue at all and we could try this out.

Looking forward to hear back.

Cheers, Andreas

Answer 2

osm4routing2 is a fast and simple way of extracting a graph from OSM .pbf files, written in Rust: https://github.com/Tristramg/osm4routing2