Self-hosting Mapbox vector tiles

Question

As presented in a talk at FOSS4G Mapbox Studio allows to create Mapbox vector tiles and export them as a .mbtiles file.

The mapbox-gl.js library can be used to dynamically style and render Mapbox vector tiles on client (browser) side.

The missing part: How can I self-host Mapbox vector tiles (.mbtiles) so that I can consume them with mapbox-gl.js?

I know that Mapbox Studio can upload the vector tiles to the Mapbox server and let it host the tiles. But that's no option for me, I want to host the vector tiles on my own server.

---

The TileStream approach below turned out to be a dead end. See my answer for a working solution with Tilelive.

---

I tried TileStream which can serve image tiles out of .mbtiles files:

My webpage uses mapbox-gl v0.4.0:

<link href='https://api.tiles.mapbox.com/mapbox-gl-js/v0.4.0/mapbox-gl.css' rel='stylesheet' />
<script src='https://api.tiles.mapbox.com/mapbox-gl-js/v0.4.0/mapbox-gl.js'></script>

and it creates a mapboxgl.Map in a JavaScript script:

  var map = new mapboxgl.Map({
    container: 'map',
    center: [46.8104, 8.2452],
    zoom: 9,
    style: 'c.json'
  });

The c.json style file configures the vector tile source:

{
  "version": 6,
  "sprite": "https://www.mapbox.com/mapbox-gl-styles/sprites/bright",
  "glyphs": "mapbox://fontstack/{fontstack}/{range}.pbf",
  "constants": {
    "@land": "#808080",
    "@earth": "#805040",
    "@water": "#a0c8f0",
    "@road": "#000000"
  },
  "sources": {
    "osm_roads": {
      "type": "vector",
      "url": "tile.json"
    }
  },
  "layers": [{
    "id": "background",
    "type": "background",
    "paint": {
      "background-color": "@land"
    }
  }, {
    "id": "roads",
    "type": "line",
    "source": "osm_roads",
    "source-layer": "roads",
    "paint": {
      "line-color": "@road"
    }
  }]
}

... with the following TileJSON specification in tile.json:

{
  "tilejson": "2.1.0",
  "tiles": [
    "http://localhost:8888/v2/osm_roads/{z}/{x}/{y}.png"
  ],
  "minzoom": 0,
  "maxzoom": 12
}

... which points to my TileStream server running at localhost:8888. TileStream has been started with:

node index.js start --tiles="..\tiles"

... where the ..\tiles folder contains my osm_roads.mbtiles file.

With this setup, I can open my webpage but only see the background layer. In the browser network trace I can see that tiles are indeed loaded when I zoom in, but the browser JavaScript error console contains several errors of the form

Error: Invalid UTF-8 codepoint: 160      in mapbox-gl.js:7

Since vector tiles are not .png images but rather ProtoBuf files, the tiles URL http://localhost:8888/v2/osm_roads/{z}/{x}/{y}.pbf would actually make more sense, but that doesn't work.

Any ideas?

Answer 1

As pointed out by @Greg, instead of TileStream (my first attempt) you should use Tilelive to host your own vector tiles.

Tilelive isn't a server itself but a backend framework that deals with tiles in different formats from different sources. But it's based on Node.js so you can turn it into a server in a pretty straight-forward way. To read tiles from a .mbtiles source as exported by Mapbox Studio, you need the node-mbtiles tilelive module.

Side note: Current Mapbox Studio has a bug under Windows and OS X that prevents an exported .mbtiles file to show up at your chosen destination. Workaround: Just grab the latest export-xxxxxxxx.mbtiles file in ~/.mapbox-studio/cache.

I found two server implementations (ten20 tile server by alexbirkett and TileServer by hanchao) who both use Express.js as a web app server.

Here is my minimalistic approach which is loosely based on these implementations:

1. Install Node.js

2. Grab the node packages with npm install @mapbox/tilelive @mapbox/mbtiles express

3. Implement the server in the file server.js:

    var express = require('express');
    var http = require('http');
    var app = express();
    var tilelive = require('tilelive');
    require('mbtiles').registerProtocols(tilelive);
   
    //Depending on the OS the path might need to be 'mbtiles:///' on OS X and linux
    tilelive.load('mbtiles://path/to/osm_roads.mbtiles', function(err, source) {
   
        if (err) {
            throw err;
        }
        app.set('port', 7777);
   
        app.use(function(req, res, next) {
            res.header("Access-Control-Allow-Origin", "*");
            res.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
            next();
        });
   
        app.get(/^\/v2\/tiles\/(\d+)\/(\d+)\/(\d+).pbf$/, function(req, res){
   
            var z = req.params[0];
            var x = req.params[1];
            var y = req.params[2];
   
            console.log('get tile %d, %d, %d', z, x, y);
   
            source.getTile(z, x, y, function(err, tile, headers) {
                if (err) {
                    res.status(404)
                    res.send(err.message);
                    console.log(err.message);
                } else {
                  res.set(headers);
                  res.send(tile);
                }
            });
        });
   
        http.createServer(app).listen(app.get('port'), function() {
            console.log('Express server listening on port ' + app.get('port'));
        });
    });
   

   Note: The Access-Control-Allow-... headers enable cross-origin resource sharing (CORS) so webpages served from a different server may access the tiles.

4. Run it with node server.js

5. Set up the webpage using Mapbox GL JS in minimal.html:

    <!DOCTYPE html >
    <html>
      <head>
        <meta charset='UTF-8'/>
        <title>Mapbox GL JS rendering my own tiles</title>
        <link href='https://api.tiles.mapbox.com/mapbox-gl-js/v0.4.0/mapbox-gl.css' rel='stylesheet' />
        <script src='https://api.tiles.mapbox.com/mapbox-gl-js/v0.4.0/mapbox-gl.js'></script>
        <style>
          body { margin:0; padding:0 }
          #map { position:absolute; top:0; bottom:50px; width:100%; }
        </style>
      </head>
      <body>
        <div id='map'>
        </div>
        <script>
          var map = new mapboxgl.Map({
            container: 'map',
            center: [46.8, 8.5],
            zoom: 7,
            style: 'minimal.json'
          });
        </script>
      </body>
    </html>
   

6. Indicate the location of the tile source and style the layers with the following minimal.json:

    {
      "version": 6,
      "constants": {
        "@background": "#808080",
        "@road": "#000000"
      },
      "sources": {
        "osm_roads": {
          "type": "vector",
          "tiles": [
            "http://localhost:7777/v2/tiles/{z}/{x}/{y}.pbf"
          ],
          "minzoom": 0,
          "maxzoom": 12
        }
      },
      "layers": [{
        "id": "background",
        "type": "background",
        "paint": {
          "background-color": "@background"
        }
      }, {
        "id": "roads",
        "type": "line",
        "source": "osm_roads",
        "source-layer": "roads",
        "paint": {
          "line-color": "@road"
        }
      }]
    }
   

7. Serve the webpage and rejoice.

Answer 2

The hosting of the vector tiles on your own is relatively straightforward. The MBTiles contains .pbf files which must be exposed to the web. That's it.

Probably easiest is to use a simple open-source server such as TileServer-PHP and put the MBTiles file to the same folder as the project files. The TileServer do all the hosting config for you (CORS, TileJSON, correct gzip headers, etc). Installation means just unpacking on a PHP enabled webserver.

If you want to start the TileServer-PHP on your laptop you can with Docker. The ready to use container is on DockerHub. Under Mac OS X and Windows it runs in a couple of minutes with the Kitematic graphical user interface: https://kitematic.com/. In Kitematic just search for "tileserver-php" and start the ready to use container/virtual machine with the project inside. Then click on "Volumes" and drop into the folder your MBTiles file. You get a running hosting for your vector tiles!

Such vector tiles could be opened in the MapBox Studio as a source, or displayed with MapBox GL JS WebGL viewer.

Technically it is even possible to host the vector tiles as a plain folder on any web server or a cloud storage, or even GitHub, if you unpack the individual .pbf out of the MBtiles container with a utility like mbutil, set the CORS, TileJSON and gzip correctly. Below is a GitHub project demonstrating such an approach as well.

Try this viewer:

and see the related repos:

• https://github.com/klokantech/tileserver-php
• https://github.com/klokantech/vector-tiles-sample
• https://github.com/klokantech/mapbox-gl-js-offline-example

Answer 3

Not to toot my own horn, but https://github.com/spatialdev/PGRestAPI is a project I've been working on that hosts .mbtiles vector tile exports from Mapbox Studio.

Still needs lots of documentation, but basically, drop your .mbtiles files into /data/pbf_mbtiles and restart the node app. It will read thru that folder and offer endpoints for your vector tiles.

It will also look thru /data/shapefiles, and create dynamic Mapbox Vector Tiles on the fly based on your .shp. You can also point at a PostGIS instance and get dynamic vector tiles.

We use them in conjunction with https://github.com/SpatialServer/Leaflet.MapboxVectorTile, a Leaflet/Mapbox Vector Tile library we've also been working on.

Answer 4

Thanks for the great question. I did not know that they had finally released a stable version of the vector tiles. Moreover, you may have to work with this answer as it is a source of ideas to your "any ideas?" question. I do not have a running studio yet.

I think one of the problems that you are running into is that you are using a tilejson file. You need a tilejson service to use a that kind of file. Hence, I believe that you need to change your sources section to an in-line URL. Try

"sources": {
"osm_roads": {
  "type": "vector",
  "url": "http://localhost:8888/v2/osm_roads/{z}/{x}/{y}.png"
 }
},

or

"sources": { 
"osm_orads": {
  "type": "vector",
  "tiles": [
    "http://localhost:8888/v2/osm_roads/{z}/{x}/{y}.png"
  ],
  "minzoom": 0,
  "maxzoom": 12
 }
},

When they use mapbox:// as the protocol, it is a alias/shorthand notation for their services. The sources section was briefly discussed around 8:40 of the video.

One step of the new vector tile process is to curate the vector data by adjusting what you want in the data. The other step is to bring the vector data back into MapBox Studio and render the data/create a style sheet. osm_roads would be step one while your c.json file is the style sheet. You may need a tile live server verses a tile stream as discussed around 15:01 of the video. The video says that you need additional meta data in the xml file.

The oddity here is that you reference the .pbf and style sheet elsewhere but the url that your provide is the resulting tiles .png files that are generated from the vector data.

You do not say, if you have a MapBox key. For your own hosting, I believe that you will have to copy the github styles and glyphs to your own server. Notice again that there is a mapbox:// protocol in glyphs tag. These two tags may not be needed because you are rendering plain lines and polygons and not POIs via icons. It is worth taking a look.

Finally, the video says that you can take a generated vector layer back into the studio to style it. You may want to reference your vector layer and apply your id:background and id:roads style there in the studio first. The video says that tile live is the server behind the scene of MapBox Studio. The idea here is to make sure that you have all the step two problems understood and fixed before you try to serve up the final vector tiles that are dynamically rendered.

Answer 5

https://github.com/osm2vectortiles/tileserver-gl-light is a lot easier to use than the main solutions mentioned -- no fiddling with JSON files needed. You just run it with

tileserver-gl-light filename.mbtiles

and then it's serving the tiles for you. It works with the predifined Mapbox GL styles such as bright-v9; after running the server you simply point whatever is consuming the tiles to

http://localhost:8080/styles/bright-v9.json

Answer 6

Every answer on this page runs a server for responding to tile requests. It's easy, cheap, and fast to host vector map tiles statically on a service like AWS S3. Maybe it would be hard to do this for a planet extract, but for the west coast US, it's worked well for me so far.

This is a really good guide for setting up AWS S3 to host tiles: https://github.com/addresscloud/serverless-tiles

Once you have the tiles from Tippecanoe or OpenMapTiles, split the .mbtiles file into a directory with mb-util:

mb-util file.mbtiles tiles_folder

Then copy them to S3 with the AWS CLI:

aws s3 cp tiles_folder s3://bucket_name/folder_path/ --recursive --content-type application/x-protobuf

Answer 7

You might want to try out our tilehut.js server. It basically does all what you need = hosting vector tiles and comes with nice examples / docs ... and is in combination with openshift it is a 5 min setup thing. Please have a look:

https://github.com/b-g/tilehut https://github.com/b-g/tilehut/tree/master/examples/simplemap_vector https://github.com/b-g/tilehut#your-own-hosted-tileserver-in-5-minutes

Answer 8

Super later, but now GeoServer serves pbf's (vector tile format)

Answer 9

Other options you can consider for tile serving;

Tegola (Written in Go)

Tiler

There's a nice introduction to this topic here

Answer 10

See Vector tiles on GCS for a Google Cloud Platform serverless solution similar to AWS S3.

Answer 11

You can now use mbtiles-s3-server (full disclosure: written by me) to serve an mbtiles file from a (versioned) S3 bucket

Taking a few bits from its README:

1. Install pip install mbtiles-s3-server

2. Copy the mbtiles file to another database with a high page size. Say your mbtiles file is mytiles.mbtiles, then create mytiles-65536.mbtiles with:

   sqlite3 my-map.mbtiles "PRAGMA page_size=65536; VACUUM INTO 'mytiles-65536.mbtiles';"
   

3. Upload this to S3, say to https://my-bucket.s3.eu-west-2.amazonaws.com/mytiles-65536.mbtiles

4. Start the server with the credentials of a user that has get GetObject and GetObjectVersion on this object

   PORT=8080 \
   MBTILES__1__URL=https://my-bucket.s3.eu-west-2.amazonaws.com/mytiles-65536.mbtiles \
   MBTILES__1__MIN_ZOOM=0 \
   MBTILES__1__MAX_ZOOM=14 \
   MBTILES__1__IDENTIFIER=mytiles \
   MBTILES__1__VERSION=1.0.0 \
   AWS_REGION=eu-west-2 \
   AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE \
   AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \
   HTTP_ACCESS_CONTROL_ALLOW_ORIGIN="*" \
     python -m mbtiles_s3_server
   

Then xyz vector tiles will then be visible at http://localhost:8080/v1/tiles/[email protected]/{z}/{x}/{y}.mvt.

There are also OpenMapTiler style files available that can point to this tileset (and so would only make sense if your vector tiles uses the OpenMapTile schema). For example http://localhost:8080/v1/styles/[email protected]/[email protected]&[email protected]

Answer 12

I'm using tile.stream to upload my datasets and get mapbox vector tiles hosted and on demand, they've a pretty good free tier.