How to use Leaflet to efficiently show parts of a massive dataset?

Question

I keep seeing non-GIS web developers running into this problem, and I'm not sure what the solution is.

1. There is some dataset of thousands of items.
2. We want to show a map to the user, with the visible subset of them shown as interactive, clickable elements.

What methods are there for doing this?

I can think of these, but they're not very satisfactory, so I'm wondering what else there is:

1. Store all the data in a GeoJSON file, transfer it to the browser, and let Leaflet display it. Problem: doesn't really work with large datasets. TopoJSON raises the limit a bit. It also causes a big delay at page load.

2. Use Mapbox, store all the data in an interactive layer on Mapbox, and use Mapbox.js to display it. Works great, but costs money, and you can't host it yourself.

3. Use GeoServer to access a PostGIS database, use the WFS-geojson leaflet plugin to access the data from there. It probably works, but the WFS-geojson Leaflet plugin doesn't seem to be maintained anymore.

4. Use CartoDB, store all the data in a CartoDB table, and use CartoDB.js to display it. Works great, but can get very expensive. It's possible to host it yourself, but installing CartoDB is non-trivial.

All of this makes me think there must be some much better, free way that I'm missing. What is it?

EDIT

Maybe I wrote off the WFS-geojson plugin too easily. There's a fork that still sees some activity (4 months ago): https://github.com/johanlahti/azgs-leaflet

Answer 1

Ok, my assumptions in 2 were incorrect. You can use mapbox.js. The end result will be a bit different, I believe - the markers themselves will be a static raster layer, but they'll be clickable.

The spec that makes large scale interactivity work is https://github.com/mapbox/utfgrid-spec

It's implemented clientside in https://github.com/danzel/Leaflet.utfgrid (leaflet plugin) and also mapbox.js.

Serverside it's implemented in https://github.com/mapbox/tilelive.js and hence TileMill eg: http://tilemill-server/tile/projectname/7/115/78.grid.json

It's also implemented in TileStache, but not tilestream or mbtiles-server. The UTFgrid data seems to be stored in the mbtiles file by TileMill, but is ignored by those.

So not only do you not need mapbox.com, you don't need mapbox.js either. Mapbox.js mostly seems to glue stuff together for convenience: a single call that instantiates a map, fetches tiles and adds interactivity.

But if you do use mapbox.js, there's one bit of the puzzle I'm missing, and that's tilejson. You give mapbox.json the tilejson file corresponding to your map.

Answer 2

There is also the leaflet-vector-layers plugin which has support for postGIS services http://jasonsanford.github.io/leaflet-vector-layers/demos/postgis-restful-web-service-framework/

By the looks of it you can filter the service.

I've used this plugin for ArcGIS services and it's been really good.

Answer 3

If you could not find the solution yet here is one: http://gis.xyz/leaflet.html#

 var owsrootUrl = 'http://217.8.255.188:8080/geoserver/opengeo/ows';

 var defaultParameters = {
     service : 'WFS',
     version : '2.0',
     request : 'GetFeature',
     typeName : 'opengeo:evernote_geom',
     outputFormat : 'text/javascript',
     format_options : 'callback:getJson',
     SrsName : 'EPSG:4326'
};

var parameters = L.Util.extend(defaultParameters);
var URL = owsrootUrl + L.Util.getParamString(parameters);

var WFSLayer = null;
var ajax = $.ajax({
    url : URL,
    dataType : 'jsonp',
    jsonpCallback : 'getJson',
    success : function (response) {
       WFSLayer = L.geoJson(response, {
            style: function (feature) {
                return {
                    stroke: false,
                    fillColor: 'FFFFFF',
                    fillOpacity: 0
                };
            },
            onEachFeature: function (feature, layer) {
                popupOptions = {maxWidth: 600};
                layer.bindPopup('<h4>'+feature.properties.url+'</h4><br>'+feature.properties.title
                    ,popupOptions);
            }
        }).addTo(map);
    }
});