Processing raw OSM data for OpenStreetMap.org

Question

Can anyone provide insight into how OSM data is processed or rendered for www.openstreetmap.org?

A specific example is that I extracted data from a recent planet.osm PostGIS dataset for an area in Missouri. The OSM data needs a lot of cleaning before it can be rendered using the correct styles. Many water bodies are stored as line strings that don't close properly, so I have to use FME for snapping and then polygon building so that I can have blue filled rivers / lakes.

If I look at the same data here the water bodies are rendered as expected.

I'm having trouble identifying all the cases where snapping is required (e.g. which 'Natural' types require it and what the tolerance should be). Also I suspect there are many other data issues that I will never see as I am dealing with all of North America.

Does everyone who downloads and uses OSM data go through their own cleanup process?

Does anyone know how this cleanup is handled by www.openstreetmap.org?

It seems like their process would be the best informed and most tested.

Here is more information on my workflow

A planet.osm file is downloaded and loaded into PostGIS, using Osmosis, into the pgsql schema. I then extract OSM xml from PostGIS for lots of small areas, again using Osmosis. Each of these small xml files is then converted into Shapefiles using FME and its broad feature categories. It is this stage (OSM xml -> Shp via FME) that I am expecting to convert lines into polygons and perform other cleanup on the data.

These Shapefiles are served up through GeoServer (and cached using GWC).

Answer 1

There are a few different angles to this, and since it's unclear how you're processing data initially, I guess I'll just give an overview.

There are two main ways to consume OSM data - by using osm2pgsql, an older utility that supports 'stylesheets' and differential updates, and Imposm, a newer, Python-based system that supports Python-based stylesheet transforms. When people do processing, a lot of it is in that kind of script. For instance, here's an imposm mapping for osm-bright, the stylesheet upon which MapBox Streets (disclosure/employee) is based.

To be more specific to what you're encountering, it's likely that you aren't properly processing osm relations properly, which, in the data model are what allow multiple linestrings to form polygons. Tools like Imposm and osm2pgsql generally handle this kind of data transformation for you.

As far as how OSM.org itself does things: edits are in a 'semantic' Postgres database, and continuously imported into a PostGIS database with osmosis, and rendered with Mapnik. There's no manual cleanup step between the database and map rendering, since the two are highly coupled and the map aims to be up-to-date.

Answer 2

In general you don't need "snapping" as such, since the original OSM data is topologically organized - a polygon (= OSM way), for example, is defined through a list of node indices (and not directly by their coordinates) - so if the starting and the ending indices are the same, that's considered a closed polygon. Otherwise, it's a polyline (like a road).

Larger bodies (like Osage river in your case) are usually defined through OSM multipolygons, which consist of a series of OSM ways (linestrings) that define the shape and holes (if any). There are several potential problems with OSM multipolygons:

1. There is more than one way to define them (just look at the specs). Different people use different rules.
2. The rules are implicit - you need to read through the OSM wiki docs to try to understand how to handle them.
3. If you use an OSM data extract, some parts of the multipolygon could be missing (since they're not geographically inside Missouri state). So you need to find a way to close the water body polygon (either by clipping it using the state boundary or manually closing it with some GUI tool).

Yes, there are other data issues, too. Mainly they stem from the very nature of OSM mapping: different people map things differently and there are no set-in-stone rules on how to do it. It's more or less a self-organized anarchy ;)

I myself never work with flattened OSM data produced by osm2pgsql - I always start with original topological data in OSM XML form and write code to process that into the form I need. But then again I don't use Mapnik for rendering, so I'm probably in the minority.

Answer 3

If you use the original database scheme from osm2pgsql, you have the related osm data models 'closed way' and 'multipolygon-relation' transformed into polygons and put in a table called 'planet_polygon'. Ways and nodes are in 'planet_line' and 'planet_point'. You can access these tables via Quantum GIS, and export them directly to shapefiles. You can also do SQL queries from inside Quantum GIS to filter the data.

I would not use osmosis for that. It does not have the polygon handling as osm2pgsql does. Osmosis stores the data in the same way the contributers deal with them (Nodes, ways and relations). It is not a suitable database scheme for rendering.