5

I want to create a difference (change file) between two Europe extracts from OSM data. The files I want to compare are:

As the files are quite large, before running on them I have tried the processing on smaller extracts:

The workflow I use is:

  • first convert xxx-180401.osm.pbf to xxx-ref.o5m, then create the diff:

osmconvert europe-180401.osm.pbf -o=eu-ref.o5m

osmconvert eu-ref.o5m europe-latest.osm.pbf --diff -o=eu-changes.o5c

Or:

osmconvert czech-republic-180401.osm.pbf -o=cz-ref.o5m

osmconvert cz-ref.o5m czech-republic-latest.osm.pbf --diff -o=cz-changes.o5c

The process works fine with the Czech Republic data. The resulting o5c file looks reasonable, its size is ~30 MB (the Czech Republic input file is ~700 MB).

I get strange results with the Europe extract. The resuling o5c file is ~40 GB, while the Europe input is ~20 GB. When inspecting the file, I have found many instances of data which was not changed on OSM in this year at all, like way 539372444.

I have also tried doing the comparison using Osmium instead of Osmconvert, but the result was the same, the change file was huge.

Am I doing something wrong, or are the Europe extracts unsuitable for the comparison for some reason?

Improve this question
5
  • 1
    Could it be due to recent removal of user id's because of the General Data Protection Regulation?
    – user30184
    May 30, 2018 at 10:17
  • 1
    +1, this was my initial thought too. Geofabrik actually has the full metadata extracts still available, but only to users that authenticate: osm-internal.download.geofabrik.de/index.html
    – Marco_B
    May 30, 2018 at 10:36
  • @user30184 Files should be compared based on version id of each node/way/relation, not on the content. Moreover, Czech Republic data from the same source work as expected. I will try converting pbf to osm sources and comparing or inspecting them manually (given the size of those files, I hope I will find some tools to handle them).
    – Suma
    May 30, 2018 at 11:16
  • You might get better results when asking at help.openstreetmap.org.
    – scai
    May 30, 2018 at 13:18
  • 1
    @scai Thanks for advice, crossposted now as help.openstreetmap.org/questions/63871/…
    – Suma
    May 30, 2018 at 13:35

1 Answer 1

Reset to default
4

As hinted in the comments and as answered on Help OSM, the GDPR data cleanup has cleaned data needed for comparison, namely version id. This seems unexpected to me, as version id does not look like personal data.

However this affects historical data only (those europe-18xxxx.osm.pbf), recent data contain this information.

To solve the issue I have downloaded the historical data from internal download location and after I have used them to create the reference o5m, the diff file size is reasonable 1 GB of 18 GB total European data.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.