4

I'm currently in the works of creating a pythonic way of extracting street names from an OSM dump for the Province of Quebec. Originally, I wanted to do this in Python to make it easier to integrate the process in future code, but I'm failing miserably, and after weeks of on and off research without getting anywhere, so now I'll honestly take what I can get.

What's the best way to extract all the street names from a dump such as these? http://download.geofabrik.de/north-america/canada/quebec.html

Here's my original code if anyone can make it work. At the moment this code maxs my 16GB of ram and 32GB of swap in about a minute before it evens begins going through each element.

from lxml import etree
import xmltodict, sys, gc
from pymongo import MongoClient

#Ultimate fix of life.
reload(sys)
sys.setdefaultencoding("utf-8")

client = MongoClient()
db = client.re
streetsDB = db.streets

hwTypes = ['motorway', 'trunk', 'primary', 'secondary', 'tertiary', 'pedestrian', 'unclassified', 'service']

#Enable Garbadge Collection
gc.enable()

def fast_iter(context, func):
    print 'Begun processing'
    placement = 1
    # http://www.ibm.com/developerworks/xml/library/x-hiperfparse/
    # Author: Liza Daly
    for event, elem in context:
        placement += 1
        print placement
        func(elem)
        elem.clear()
        while elem.getprevious() is not None:
            del elem.getparent()[0]
    del context

def process_element(elem):
    data = etree.tostring(elem)
    data = xmltodict.parse(data)
    keys = data['way'].keys()
    if 'tag' in keys:
        if isinstance(data['way']['tag'], dict):
                    if data['way']['tag']['@k'] == 'highway':
                        if data['way']['tag']['@v'] in hwTypes:
                            streetsDB.insert(data)
        else:
            for y in data['way']['tag']:
                if y['@k'] == 'highway':
                    if y['@v'] in hwTypes:
                        streetsDB.insert(data)
                        break
    del data
    del keys
    gc.collect()

context = etree.iterparse('quebec-latest.osm', tag='way' )
fast_iter(context,process_element)
1

2 Answers 2

8

I would simply use OGR's OSM Driver.

With this script you get a list of all the street names. I just tried it with a small sample dataset. I don't know how it will perform with a bigger one.

import ogr

ds = ogr.Open('map.osm')
layer = ds.GetLayer(1) # layer 1 for ways

nameList = []
for feature in layer:
    if feature.GetField("highway") != None: 
        name = feature.GetField("name")
        if name != None and name not in nameList: 
            nameList.append(name)

print nameList

You should not experience any RAM problems. If the database is above 100 MB, it will be written in a temporary file on disk.

3
  • namelist should be a set
    – Elazar
    Nov 9, 2014 at 12:00
  • What would be the advantage of that?
    – ustroetz
    Nov 9, 2014 at 17:41
  • Asymptotic complexity. (And is not None is better). It can be simply nameList = {feature.GetField("name") for feature in layer if feature.GetField("highway")}-{None}
    – Elazar
    Nov 9, 2014 at 19:57
3

I'm not exactly sure what your stack is, but I'd do this:

  1. Load extract into PostGIS using osm2pgsql
  2. Run a query like:

    SELECT name FROM planet_osm_line WHERE highway in ('motorway', 'trunk', 'primary', 'secondary', 'tertiary', 'pedestrian', 'unclassified', 'service')

(You're missing 'residential','living_street', 'track', plus all the '*_link' types, btw.)

If you're not using that stack, this script (install.sh) should help you get started: https://github.com/stevage/tilemill-server

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.