I'm new to the fascinating GIS world, and I've been researching the whole topic for the past few days. My goal is to serve planet-wide maps with OSM data on my own servers.
My stack is:
- Postgres + Postgis - For storing, transforming and processing GIS data.
- Mapnik - Library to render tiles
- Tilestache - Serve and cache tiles
- Nginx - Reverse proxy to Tilestache
- Leaflet (Slippy map client)
In order to import and style OSM data, I'm using
- PBF file (Extract/Planet)
- osm2pgsql
- CartoCSS
- openstreetmap-carto
- Kosmtik - Map GUI for development/debugging
Since the full planet file is huge, I'm currently using a extract of Sao Paulo, which is good/small enough for testing.
Before I reach to the problem I'm having, I'll guide through the steps I took so far:
What worked
PostgreSQL and PostGIS are properly installed. I'm using 9.6 for the former, 2.3 for the latter. I've also added the postgis_topology
and hstore
extensions, which will be used later.
I have osm2pgsql 0.92.0 installed. In order to import the Sao Paulo extract, I use
osm2pgsql -G U <user> -d <db> -C 1000 -W --hstore --style openstreetmap-carto.style --tag-transform-script openstreetmap-carto.lua <pbf>
The hstore
, style
and tag-transform-script
arguments are required for the proper utilization of OSM's style, as described here.
As described on the openstreetmap-carto installation, I've also added custom indexes and downloaded the required shapefiles and fonts (except for the emoji ones).
I verified the extract data is correctly loaded on the database by using QGIS. I was able to view and query with all points, polygons, lines and roads. It's all there.
Problems
The next step is rendering the tiles, and that's where I'm having trouble. I have mapnik and python-mapnik 3.x, as well as carto 0.18.2.
From the OSM's project.mml, I've generated my own project.xml using carto
. I've made the following changes to the result:
- Add the Postgres username (Parameter
user
on Mapnik's XML). Remaining connection info is grabbed from ~/.pgpass. - Replace relative directory
data/<shape>.shp
to/full/path/data/<shape>.shp
- Replace relative directory
symbols/<symbol>.svg
to/full/path/symbols/<symbol>.svg
I sort of unit-tested these changes: incorrect directories for shapefiles and symbols will raise an error when Mapnik tries to load the style. Same applies for incorrect DB information.
And here's the problem: every tile rendered by Mapnik is "blank" (blue-ish background, which is the defined background for OSM). Here's what I've tried to fix this problem:
Things I've tried
Maybe the generated style is invalid somehow. Maybe some polygon/shape (the ocean's?) is covering another one.
I've installed Kosmtik, and it worked fine! I was able to browse through the extract, inspect data, zoom in and out on all levels.
Maybe I am generating tiles at the wrong location
I've set up TileStache with Mapnik as the provider. Then, I've set up redirect rules from http://[abc].tile.openstreetmap.org
to http://myserver/<layer>
using switcheroo, as described here. This means that accessing openstreetmaps.org
would have all tiles rendered on my server.
Interestingly, zoom 1 works (the one with 4 tiles). I suppose it only uses the planet/world shapefile, which was downloaded with the style. All else, regardless of the zoom or location, renders to "blank". So even when browsing at the extract's location, it won't render correctly.
Note that, when using Kosmtik, I was able to see the extract's name from as low as zoom 4.
Maybe I am generating tiles at the wrong location
Projections are hard. Maybe there is something being "mistranslated" somewhere.
So I analyzed the requests Kosmtik was making (since these were working). It uses the Modestmaps library, which translates to Slippy format (<zoom>/<x>/<y>
), same format used by Leaflet, TileStache and openstreetmap.org
.
For instance, requesting 17/48561/74357.png
on Kosmtik works, but the same tile returns as "blank" on TileStache. My theory was that somehow TileStache was translating it to another coordinate, outside the extract area. (Even though TileStache was using "spherical mercator" as the projection).
So I've decided to use Mapnik's python bindings directly. So far I've handled three different types of projections, here's what I know about them. Please correct me if I'm wrong:
- EPSG4326/WGS84 - Usual lon/lat projection, same as used on GPS. Unit is degrees.
- EPSG3857/{Web,Google,Pseudo} mercator/900913 - Used on most tiled web maps. Unit is in meters (not "real" meters, as it distorts heavily near the poles).
- "Slippy format" - Not a projection, but yet another format to translate.
So, the Slippy 17/48561/74357
would be near -46.632(lon), -23.531(lat)
on EPSG4326, and near -5191050.49, -2696361.67
on EPSG3875/Google mercator.
Using python, I've generated these coordinates' tiles directly at Mapnik. The result was always blank tiles. I believe I'm using the correct projections (more on this later).
First, one thing to note. It's clear the problem is not in Mapnik, nor the style. For one, Kosmtik uses Mapnik as backend, and it works fine. Second, if I purposely generate a tile outside the extract area on Kostmik, I get a blank tile, as expected.
This leads me to believe I'm using the wrong projection at Mapnik's python script and TileStache. Before showing the code, one remark about OSM (again, please correct me if I'm wrong):
- OSM "native" data is stored in EPSG4326
- osm2pgsql imports OSM data as EPSG3857, and that's the projection on Postgis.
- Mapnik's style (XML) has two projections, input and output, explained here:
- Input projection is defined at the Map object, and is the "coordinate system in which the map is rendered"
- Output projection is defined at the Layer object, and must be the same as my data.
My resulting Mapnik's XML has:
<Map srs="+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0.0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs +over" background-color="#b5d0d0">
#snip#
<Layer name="world" #snip# srs="+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0.0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs +over">
So, the way I see it, both the input and output projections are in "merc"(3857), and that's correct! My data is stored with 3857, and the resulting map is expected to be at 3857.
Finally, here are the different Mapnik calls I've tried:
Box2d with 3857 coordinates:
map = mapnik.Map(800, 800)
bbox = Box2d(-5191050.49, -2696361.67, -5191059.49, -2696370.70)
mapnik.load_map(map, '/path/to/style.xml')
map.zoom_to_box(bbox)
mapnik.render_to_file(map, 'output.png') # Blank
Box2d with 4326 coordinates transformed to 3857
merc = mapnik.Projection('+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +no_defs +over')
longlat = mapnik.Projection('+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs')
bbox = mapnik.Box2d(-23.53, -46.63, -24.03, -47.03)
transform = mapnik.ProjTransform(longlat, merc)
merc_bbox = transform.forward(bbox)
# load, zoom, render -> Blank
I've also tried with different styles. Result is always blank, but that's unexpected because I believe these coordinates would be within my extract. When trying to render coordinates outside the extract, it correctly renders as blank.
Anyway, I'm lost and I was wondering if anyone could shed some light, I must be missing something. But so far, I've been battling with this issue for well over 24 hours and I have no idea what could be wrong.