After weeks of research and headache, I finally figured out the correct architecture of the local OSM server. My problem was that the employee who created the local OSM server hasn't actually finished his work (all he has done is cloning the open source OSM project and deploying it on the local server).
Probably, the following information won't be helpful for those who have been engaged in OSM for a long time, but if I found this information a couple of weeks ago, I would be happy. So I’ll try to explain what I've done in order to save someone's time and nerves.
Architecture of the local OSM server
Here it is. All the architecture should be deployed on your local machine.
Local OSM server
Local OSM server is an open source Ruby on Rails project, which source code is available on Github. Everyone can clone this project and start the Rails server on his local machine, but, by default, all the geographical data (countries, cities, ways, buildings) will be taken from the main OSM server.
However, this project creates a local PostgreSQL database (main database), where all changes from the edit mode are saved. That's the problem: you draw geographical objects in the edit mode and they are saved to the local main database, while in the view mode the local server gets data from the main OSM server's main database. But we'll fix it.
Mapnik/Renderd server
Tiles are small (256px * 256px) png images of which the view mode map consists. They should be rendered accordingly to your rendering database. The rendering database is a separate database which stores data needed for rendering - generating tiles. The rendering database data should be synchronized with the local main database.
After cloning the openstreetmap-website project and running it on your local machine, you will see a map with normally rendered tiles. But these tiles do not come from your local rendering database, they come from the main OSM server's rendering database. Moreover, the rendering database doesn't come with the openstreetmap-website project - you have to create it manually and also add data there.
Mapnik/Renderd server is used to generate tiles according to data from the local rendering database. It also doesn't come with the openstreetmap-website project.
That's why you can't see the edit mode's changes in the view mode - because you have neither the rendering database, nor the local Mapnik/Renderd server. The local map can't get tiles from nowhere, so it gets them from the main OSM server. Don't worry, I'll explain how to fix it.
Nominatim server
Nominatim server is used to search address by query or coordinates. It uses its own Nominatim database. By default, openstreetmap-website uses the main OSM server's Nominatim, too. And you should deploy your own Nominatim server on your local machine.
How to create a local OSM server
Directories structure
Very soon you will have to download a lot of repositories to your machine, so organize them correctly. E.g. I have placed all OSM repositories in one directory:
ls /home/rion/openstreetmap
mapnik Nominatim openstreetmap-website osmosis
mod_tile openstreetmap-carto osm2pgsql
Local OSM server deployment
Follow the official installation instruction step by step to clone and deploy a local OSM server. The only notes that I have here are for the moment when you create a main database' user. First, his <username> should be the same as your Linux user's username (rion in my case). Second, after creating the user, give him a password (it will be needed soon):
sudo -u postgres -i
psql
alter user rion with password 'password';
\q
exit
After you have deployed the local OSM server, PostgreSQL openstreetmap database will be created. You always can check a full list of databases with these commands:
psql postgres
\l
\q
Download your country's geographical data dump in pbf format from here. E.g. I have downloaded a http://download.geofabrik.de/asia/japan-latest.osm.pbf file. Then you will upload it to your main database.
Follow the official configuration instruction to upload data to the main database and create the edit mode users. My osmosis command to upload data to the main database was:
osmosis --read-pbf japan-latest.osm.pbf --write-apidb host="localhost" \
database="openstreetmap" user="rion" password="password" \
validateSchemaVersion="no"
Have a cup of tea (a lot of cups?), because uploading data to the main database will take some time (it depends on pbf file size).
After you have uploaded data to the main database and created the edit mode user(s), you can finally test the edit mode. Here I had an issue. After saving the edit mode changes via iD editor, I had 404 error: land.html is missing. I have fixed it by copying land.html file to the openstreetmap-website/public folder.
Mapnik/Renderd server deployment
- Use this Ubuntu 18.04 instruction (or this Ubuntu 16.04 instruction) to deploy the Mapnik/Renderd server on your local machine. I haven't created a
renderaccount user and have used the already created user instead (rion in my case). Use the same pbf file to populate the rendering database.
- At this moment, you should be able to get tiles from, e.g.
http://localhost/hot/0/0/0.png. If you did everything the same as in the instruction, generated tiles' metadata will be stored in /var/lib/mod_tile/ajt directory (note that tiles are stored as meta files, not png).
All you need to do here is to modify the openstreetmap-website source code to get tiles from the local server, not from the main OSM server. By default, tiles come from *.tile.openstreetmap.org. So search for tile.openstreetmap.org in the whole project and replace it to your localhost. I did it using RubyMine IDE's Ctrl + Shift + F shortcut. E.g.:
"https://a.tile.openstreetmap.org/${z}/${x}/${y}.png" # Replace this
"http://localhost/hot/${z}/${x}/${y}.png" # With this
- Restart the local OSM server. Now your server uses tiles from localhost. If you have downloaded only one country's
pbf, then you can notice that the view mode map shows only that country's data. Note that your browser can cache old tiles came from main OSM server, if so, then press Ctrl + F5 to load the map without cached tiles.
Nominatim server deployment
- Install Nominatim using the official Ubuntu 18.04 instruction (or Ubuntu 16.04 instruction). Here I haven't created a new
nominatim user, too. I'll use the already created rion user further.
Populate the Nominatim database with the previously downloaded pdb file:
cd /home/rion/openstreetmap/Nominatim/build/utils
./setup.php --osm-file /home/rion/openstreetmap/japan-latest.osm.pbf --all
- This operation will also take some time. After finishing it will create a
nominatim PostgreSQL database.
- Now you can query the local Nominatim. E.g.:
http://localhost/nominatim?q=tokio&format=json.
Here you need to switch the openstreetmap-website to the local Nominatim. Replace the https://nominatim.openstreetmap.org url with http://localhost/nominatim in files:
config/settings.yml
lib/nominatim.rb
- Restart the Rails server to make the changes live.
Sync the databases periodically
It's cool that you have deployed the main, rendering and nominatim databases. But all map changes that you make in the edit mode are saved only to the main database. And they won't magically appear in the rendering and nominatim databases. So we need to periodically supply the main database' changes to the rendering and nominatim databases. Also we need to re-render tiles that have been touched by these changes.
To do it I wrote bash scripts, which will be regularly launched by cron. The logic is:
Before you create your own adaptation of sync_osm_databases.sh, get a value of your PATH environment variable. Then use it in your sync_osm_databases.sh.
printenv | grep PATH
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/rion/openstreetmap/osmosis/bin/
A path to osm2pgsql also may be different in your system, so run the following command and use the path to osm2pgsql in your sync_osm_databases.sh:
whereis osm2pgsql
osm2pgsql: /usr/local/bin/osm2pgsql
sync_osm_databases.sh:
#!/bin/bash
# Use your own PATH environment variable here.
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/rion/openstreetmap/osmosis/bin"
# Go to the main OSM directory where the OSM itself and all its tools are located.
cd /home/rion/openstreetmap
# Delete old temp files if they are present.
rm sync_osm_databases/main-db-changes.xml
rm sync_osm_databases/expired-tiles.list
# Get the current and the 1 hour ago dates.
# They will be needed to export main database changes for the last hour.
NOW=$(date -u +"%Y-%m-%d_%H:%M:%S")
HOUR_AGO=$(date -u -d '1 hour ago' +"%Y-%m-%d_%H:%M:%S")
# Write all main database changes for the last hour to xml.
osmosis --read-apidb-change host="localhost" database="openstreetmap" user="rion" password="password" intervalBegin=$HOUR_AGO intervalEnd=$NOW validateSchemaVersion="no" --write-xml-change file="sync_osm_databases/main-db-changes.xml"
# Load the changes to the rendering database.
/usr/local/bin/osm2pgsql -d gis --append --slim -G --hstore --tag-transform-script openstreetmap-carto/openstreetmap-carto.lua -C 2500 --number-processes 1 -S openstreetmap-carto/openstreetmap-carto.style sync_osm_databases/main-db-changes.xml
# Update & index Nominatim database.
Nominatim/build/utils/update.php --import-diff sync_osm_databases/main-db-changes.xml
Nominatim/build/utils/update.php --index
# Create a list of tiles that needed to be re-rendered.
/usr/local/bin/osm2pgsql -H localhost -d gis -U rion -W -S openstreetmap-carto/openstreetmap-carto.style -s -C 2500 -a --number-processes 1 sync_osm_databases/main-db-changes.xml -e16 -o sync_osm_databases/expired-tiles.list
# Expire these tiles, so they will be re-rendered.
cat sync_osm_databases/expired-tiles.list | mod_tile/render_expired --map=ajt --min-zoom=10 --max-zoom=20 --delete-from=10 >/dev/null
sync_osm_databases_task.sh:
#!/usr/bin/expect -f
set timeout -1
# Launch the sync_osm_databases.sh script. Use the absolute path here.
spawn /home/rion/openstreetmap/sync_osm_databases/sync_osm_databases.sh
# Enter the database password when the spawned script asks it.
expect "Password:"
send -- "password\r"
expect eof
After you have created these bash scripts, create a new cron task via crontab -e (do not use sudo crontab -e, because the root user won't be able to access the databases via the previously created database user's credentials). Add the following line(s) to your cron schedule.
crontab -e:
# A task for testing purposes. Every time cron runs this task,
# its output will be printed in cron.log file, so if something will go wrong,
# you can see what exactly and fix it.
# But the file won't be recreated every time, logs will be added to its end.
# So it's better to disable logging after testing to avoid the file's overgrowth.
# After testing you can delete this line and uncomment the next one.
#
# The cron task will be launched every hour in 0 minute.
# You can replace 0 with any 0-59 digit to launch the task in other minute
# (it's convenient for testing).
0 */1 * * * /home/rion/openstreetmap/sync_osm_databases/sync_osm_databases_task.sh >> /home/rion/openstreetmap/sync_osm_databases/cron.log 2>&1
# A task for production. Uncomment it after testing and deleting the above one.
# 0 */1 * * * /home/rion/openstreetmap/sync_osm_databases/sync_osm_databases_task.sh
After scheduling the cron task, you can draw a house in the edit mode of your OSM server. The syncing script will be launched in 0 minute of the next hour. It will take a few minutes to complete, so your house will appear a few minutes later than 0 minute of the next hour.
Luckily, that's all!
