Updating Overpass-API database with local changes file

Question

I have a local OSM server and its data is stored in openstreetmap PostgreSQL database. Also I have Overpass-API installed for the local OSM server (using this tutorial).

The openstreetmap database is sometimes updated by changes that users of the the local OSM server make in edit mode (by iD editor). But Overpass-API uses its own database, so I need to periodically sync these two databases.

I have already populated the Overpass-API database with openstreetmap database' data:

cd /home/rion/openstreetmap

# Dump all data from the openstreetmap database.
osmosis --read-apidb host="localhost" database="openstreetmap" \
user="rion" password="password" validateSchemaVersion="no" \
--write-xml file="main-database-dump.xml"

# Compress this dump.
bzip2 -c main-database-dump.xml > main-database-dump.xml.bz2

# Specify paths to Overpass-API's executable, database, replicate files.
EXEC_DIR="/home/rion/openstreetmap/Overpass-API/build"
DB_DIR="/home/rion/openstreetmap/Overpass-API/build/db"
REPLICATE_DIR="/home/rion/openstreetmap/Overpass-API/build/replicate"

# Upload the dump to Overpass-API database.
./Overpass-API/src/bin/init_osm3s.sh main-database-dump.xml.bz2 $DB_DIR $EXEC_DIR

---

So, after some changes made in the openstreetmap database, I need to upload them to Overpass-API database, too. I use this command to pull out the openstreetmap database' changes to xml file:

# Specify dates between which I will read openstreetmap database' changes.
NOW=$(date -u +"%Y-%m-%d_%H:%M:%S")
HOUR_AGO=$(date -u -d '1 hour ago' +"%Y-%m-%d_%H:%M:%S")

# Read openstreetmap database' changes between $HOUR_AGO and $NOW dates.
osmosis --read-apidb-change host="localhost" database="openstreetmap" \
user="rion" password="password" intervalBegin=$HOUR_AGO \
intervalEnd=$NOW validateSchemaVersion="no" \
--write-xml-change file="main-database-changes.xml"

# Compress the changes file.
bzip2 -c main-database-changes.xml > main-database-changes.xml.bz2

So now I have the compressed xml changes file and need to upload it to the Overpass-API database. So the question is: how to do it?

I know that there is a ./Overpass-API/build/bin/update_database util, but how to use it? I've tried to use it this way:

./Overpass-API/build/bin/update_database --db-dir=$DB_DIR \
main-database-changes.xml.bz2

But it gives me an error:

Unkown argument: main-database-changes.xml.bz2

Usage: update_database [--db-dir=DIR] [--version=VER]
[--meta|--keep-attic] [--flush_size=FLUSH_SIZE]
[--compression-method=(no|gz)] [--map-compression-method=(no|gz)]

How do I use the update_database util with a local changes file or are there some other utils to do it?

Answer 1

Regarding the error message: you need to provide an OSM file in XML format (not xml.bz2 !) via standard input:

update_database --db-dir=$DB_DIR < main-database-changes.xml

Please note that update_database can only be used for the initial database setup(!). To apply subsequent changes, you have to use the update_from_dir binary instead to apply your osc files

Take a look at https://github.com/drolbr/Overpass-API/blob/master/src/bin/fetch_osc_and_apply.sh how this is implemented. Take particular note of the --flush-size=0 parameter, and the --version parameter, which is used to provide a meaningful timestamp in your query response.

Answer 2

Okay, I couldn't understand how to update Overpass-API database via dispatcher, so I have used the following workaround. I've written a bash script which will be executed every midnight by cron. I know that it's not a good way, but it works, at least. Maybe someone will post here a better solution.

Bash script (sync_overpass.sh)

#!/bin/bash

# Need this PATH variable to use Osmosis in a cron task.
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/rion/openstreetmap/osmosis/bin"

cd /home/rion/openstreetmap

# Remove an old openstreetmap changes file if it exists.
rm sync_osm_databases/main-database-daily-changes.xml

# Specify paths to Overpass-API executable & database files.
EXEC_DIR="/home/rion/openstreetmap/Overpass-API/build"
DB_DIR="/home/rion/openstreetmap/Overpass-API/build/db"

# Specify dates between which I will read openstreetmap database' changes.
NOW=$(date -u +"%Y-%m-%d_%H:%M:%S")
DAY_AGO=$(date -u -d '1 day ago' +"%Y-%m-%d_%H:%M:%S")

# Get the running dispatcher's pid.
DISPATCHER_PID=$(pgrep -f Overpass-API/build/bin/dispatcher)

# Get openstreetmap database' daily changes.
osmosis --read-apidb-change host="localhost" database="openstreetmap" \
user="rion" password="password" intervalBegin=$DAY_AGO intervalEnd=$NOW \
validateSchemaVersion="no" --write-xml-change \
file="sync_osm_databases/main-database-daily-changes.xml"

# Kill the dispatcher and delete its lock files.
kill $DISPATCHER_PID
rm $DB_DIR/osm3s_v0.7.54_osm_base
rm /dev/shm/osm3s_v0.7.54_osm_base

# Upload changes to Overpass-API database.
$EXEC_DIR/bin/update_database --db-dir=$DB_DIR --meta=no \
< sync_osm_databases/main-database-daily-changes.xml

# Start the dispatcher.
nohup $EXEC_DIR/bin/dispatcher --osm-base --db-dir=$DB_DIR --meta &

cron task:

0 0 * * * /home/rion/openstreetmap/sync_osm_databases/sync_overpass.sh