How to deal with versioning in OpenStreetMap?

Question

The topic of managing geospatial data in a more general sense has come up before here. The topic of versioning was mentioned there as well, but not really dealt with.

Traditional geospatial data collection and maintenance only need to deal with versioning internally, as the database is only updated from within the organization. This is not the case in crowdsourced geodatabases like OpenStreetMap. There, anyone can come along and add, modify or delete objects. In OpenStreetMap this is dealt with in a rudimentary way: each object has an integer version number, and only the object with the highest version is exposed in the live database. The database uses optimistic locking, so users must resolve all conflicts that occur when uploading contributions manually.

This all works reasonably well as long as human contributions through the editors (JOSM, Potlatch) are the only mode of contribution - but they aren't. Increasingly, imports of open public sector data are conducted. These make for more complex versioning issues. Consider the following scenario:

1. A building object is being imported from an open public sector dataset
2. The building receives some modifications by human contributors (attributes, geometry, or both)
3. A new version of the public sector data becomes available and is imported.

Currently, in step 3. the human contributions would be lost, unless each building that received community modifications is manually merged with the new import.

How can OpenStreetMap deal with this situation? Do we need to look at distributed version control in software development? How can methods of DVC be adapted to deal with distributed spatial data maintenance?

Answer 1

I've dreamed of someone implementing non-destructive editing for GIS data. It's compute intensive but shouldn't be difficult implement in an RDBMS.

Start with a snapshot of the data. Any changes are saved as edits, the original data remains unchanged. In your example, the buildings come in initially from the public sector data. When a user makes an edit, the change or difference is saved in a separate table. When someone views the feature they are given the original plus any edits applied. Subsequent edits are the computed difference between the new feature shape and original plus all previous edits.

This gives you ability to undo at a fine-grained level. It is essentially what version control does. A good example of non-destructive editing is Apple's Aperture. Imported digital images in Aperture are not modified directly. Changes in levels, sharpening, blur, etc. are stored as edits and applied on the fly when you work with an image. Any change can instantly be removed.

Of course, you would take snapshots of the DB for distribution and use in production environments. This would only be for development and editing.

Take a look at Versioning PostGIS, pgVersion and Post Facto for ideas and possible solutions. These are version control systems implemented in PostgreSQL databases.

Answer 2

OSM uses Postgres and Postgis which keeps a snapshot of the database.

To implement this on your own server and database

http://wiki.openstreetmap.org/wiki/Databases#Choice_of_DBMS

Database (plantet.osm) is updated weekly http://wiki.openstreetmap.org/wiki/Planet_dump

Osmosis is used to "it has components for reading from database and from file, components for writing to database and to file, components for deriving and applying change sets to data sources"

http://wiki.openstreetmap.org/wiki/Osmosis

Changsets: http://wiki.openstreetmap.org/wiki/Osmosis/Detailed_Usage#Changeset_Derivation_and_Merging

Answer 3

I though of this problem and had an idea, but not tested it. It might work:

Use version control system like Mercurial or Git. Mercurial will be easier, since it allows to easily create anonymous branches.

Now, from the initial revision start a branch for the public dataset imports. So, there'll be 2 branches:

1. mainline (OSM)
2. public dataset X

An import from the public dataset should be done in branch 2, then merged into OSM branch.

Your scenario might work like this:

• an object did not exist
• then it is imported and merged into branch 1
• then it is modified in mainline, including the geometry
• it's imported again in branch 2
• when it's merged into branch 1, only the data that was updated in branch 2 is updated in branch 1

This might require splitting the data into multiple files, one per object and probably into a format like json, so that VCS can easily deal with changes to separate attributes.

{
     id: 1357
     lat: 1,
     lon: 2,
     tags: {
          'building': 'entrance'
     }
     type: 'node',
}
{
     nodes: [
         1357,
         2468
     ],
     tags: {
         building: 'yes',
     }
     type: 'way',
}

I know that splitting an information in a billion files is too much for any system. Instead the core of VCS should be used and OSM data should be processed and fed into VCS in a versionable form. (Or a filesystem can be mocked).

I can't guarantee this will work.