3

I found that two exactly identical Geopackages do not share the same checksum as one would probably expect (and is vital when syncing files using rsync for example). Here is a reproducible example:

Get some sample data:

$> wget https://biogeo.ucdavis.edu/data/gadm3.6/gpkg/gadm36_ZWE_gpkg.zip
$> unzip gadm36_ZWE_gpkg.zip

Now use ogr2ogr to copy the data to two different files with exactly the same parameters:

$> ogr2ogr -f GPKG test1.gpkg -nln testlayername gadm36_ZWE.gpkg
$> ogr2ogr -f GPKG test2.gpkg -nln testlayername gadm36_ZWE.gpkg

If we check the checksums they happen to differ:

$> md5sum test*.gpkg
ad2fe21516a94ab9f2c0bca37c171028  test1.gpkg
240b65bac70687d8fc65c70c9c4dbb46  test2.gpkg

So after some experiments, I found that it must have to do something with the "Geo" component, as

ogr2ogr -f sqlite test[1|2].sqlite -nln testlayername gadm36_ZWE.gpkg

would generate the same chechsum for both outputs while

ogr2ogr -f sqlite test[1|2].sqlite -nln testlayername gadm36_ZWE.gpkg -dsco "SPATIALITE=YES"

would not generate the same chechsum for both outputs.

Am I completely off here? Does anybody know what causes the difference in checksums and is there any way to prevent that when using GPKGs?

Improve this question
2
  • 2
    I would imagine there is a timestamp somewhere in the metadata that differs
    – Ian Turton
    Commented Jan 28, 2021 at 16:08
  • vbindiff is a nice tool to look at differences between files on a binary level (includes a string-y view of readable text as well).
    – bugmenot123
    Commented Jan 31, 2021 at 14:37

2 Answers 2

Reset to default
3

As well pointed out @Ian Turton, probably there is a different timestamp in the metadata of both files created with ogr2ogr command. For creating data copies you only should use operating system commands. For testing there is nothing wrong with the integrity of files manipulated in this way, I downloaded twice same file in two different folders and run md5sum command as follows. Results were identical.

zeito@zeito-pc:~/Desktop/folder1$ md5sum gadm36_ZWE_gpkg.zip
bc6fe187ed2bb0f1ccdc5d525787a134  gadm36_ZWE_gpkg.zip
zeito@zeito-pc:~/Desktop/folder2$ md5sum gadm36_ZWE_gpkg.zip
bc6fe187ed2bb0f1ccdc5d525787a134  gadm36_ZWE_gpkg.zip

Afterward, I copied gadm36_ZWE_gpkg.zip (from folder2) to my Desktop and ran md5sum again. Result was also identical.

zeito@zeito-pc:~/Desktop/folder2$ cp gadm36_ZWE_gpkg.zip /home/zeito/Desktop
zeito@zeito-pc:~/Desktop/folder2$ cd ..
zeito@zeito-pc:~/Desktop$ md5sum gadm36_ZWE_gpkg.zip
bc6fe187ed2bb0f1ccdc5d525787a134  gadm36_ZWE_gpkg.zip
Improve this answer
1
  • Thanks for your input on that. It helped me to get to the bottom of this. In fact the use of ogr2ogr was only used to provide a minimal and reproducible example. I added an answer going around the whole timestamp issue.
    – loki
    Commented Jan 31, 2021 at 12:43
1

Indeed, as @IanTurton pointed out in his comment, a time stamp is recorded in the geopackage by default. This is what's causing the difference in the checksum. So, if you can discard this information, you can just overwrite this information and get the same check sums:

$> sqlite3 test1.gpkg "update gpkg_contents set last_change = '2000-01-01T01:01:01.000Z'"
$> sqlite3 test2.gpkg "update gpkg_contents set last_change = '2000-01-01T01:01:01.000Z'"

$> md5sum test*.gpkg
69dd7716fcaf505557d47f5d6bffd2aa  test1.gpkg
69dd7716fcaf505557d47f5d6bffd2aa  test2.gpkg

However one has to keep in mind, that even opening the gpkg in e.g. QGIS will overwrite the timestamp and in consequence manipulate the checksum again.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.