Programatically Cache WMTS for Offline Use

Question

Background

I have developed an app for iOS using the Swift language. The app uses online map services, including WMTS, WMS, TMS, XYZ and makes heavy use of OpenLayers to display these. I would like to be display some of these maps offline and therefore need to be able to cache selected portions of them locally (ie, selected extent and selected range of zoom levels).

(Don't need to remind me about copyright or licensing issues - I have my own Geoserver service so it's fine for those maps at least, and the app will have warning to users of other map services.)

TMS Works

I can cache a portion of a TMS map service, and it works fine offline displayed from the cache. I essentially calculate which tiles are required, download them, and store them on the file system in the same heirarchy as they are in their online URLs. Then they can simply be called by substituting the "http://servername/.../" with "file:///.../" in the base service URL.

WMTS is Complicated

However, I'm having trouble doing the same thing for WMTS, which has a much more complicated Capabilities.xml. Additionally, WMTS GetTile URLs are not well suited to offline caching as some of them could result in a flat directory structure and some of them can include characters that are inappropriate for filenames. So if I attempted to replicate a similar method to what I'm doing with TMS, all the tiles would end up in the same directory, making for very slow file system traversal and therefore inefficient reading of the tiles, plus I would have to work out some scheme for translating the file paths AND the Capabilities to avoid certain characters that don't play well with file names (eg, ":" is often used in WMTS tile paths, but is forbidden in iOS file names - if you see it in a file name, it is actually using a different character in the real file name, behind the scenes).

Proposed WMTS Work-Around

In order to work around these issues, I'm attempting to produce an offline TMS cache for the online WMTS service. This enables me to work with simple file names and a directory tree that doesn't result in excessive numbers of files in any one directory. I should then be able to feed the TMS cache to OpenLayers in the same way that I already do for my cached TMS services.

This is almost working. It downloads all of the right tiles, but the row numbers, column numbers, extent and origin are all different to what I get when I download from the equivalent TMS service from the same Geoserver, and the cached map doesn't display in OpenLayers.

Can anybody provide advice on how to convert online WMTS to an offline cache that I can feed into OpenLayers?

As an example, just one of the issues I'm struggling with is... how to convert WMTS' array of DIFFERENT origins ("TopLeftCorner") into a single TMS origin? (The origin-X-coordinate seems to be same as for TMS, but none of the WMTS origin Y-coordinates are anywhere near the TMS origin-Y coordinate for the same Geoserver map layer!)

I'd like to provide some code to show what I'm trying so far, but the nature of the beast is so complicated that it would require a prohibitively large base of code to be posted (and how many GIS people know Swift anyhow?).

Answer 1

I now have a working solution for which I can use exactly the same implementation for WMTS, TMS and XYZ - which is great to avoid three different methods for the three different service types.

This solution is implemented mostly in OpenLayers3 (JavaScript) (which my app already uses for map rendering), with some chatter between Swift and Javascript to pass data structures back and fourth as required.

Here's the relevant OpenLayers (JavaScript) parts of the solution:

1. Query OpenLayers to generate a data structure that includes all possible tile coordinates (zoom, X, Y) for the specified extent and zoom levels, as well as the URLs used to fetch each of those tiles:

function getRasterServiceTileURLs(layerIndex, zoomMin, zoomMax, extent) {
    var tileURLs = []
    var source = rasterServiceLayers[layerIndex].getSource();
    var tileGrid = source.getTileGrid();
    var urlFunc = source.getTileUrlFunction();
    for ( zoom = zoomMin; zoom <= zoomMax; zoom++ ) {
        tileGrid.forEachTileCoord(extent, zoom, function(tileCoord) {
            var url = urlFunc.call(source, tileCoord, ol.has.DEVICE_PIXEL_RATIO, source.getProjection());
            tileURLs.push({"tileCoord": tileCoord, "url": url});
        });
    }
                                      
    return tileURLs;
}

2. Download the images for all of those URLs and store them in directory/file structure that can be used later to find each tile by its z/x/y reference (I used a TMS-style directory structure like: ".../{z}/{x}/{y}.ext"). (My app does this in Swift using a multi-threaded background operation queue).

3. Query OpenLayers to get the layer source's "tile grid" structure, and save this, the layer's projection and the extent for which tiles were captured somewhere with the tiles (I saved these in a "manifest.plist" file in the top-level directory along with some other metadata; the tileGrid object was saved as a JSON string):

function getRasterServiceTileGrid(layerIndex) {
    return rasterServiceLayers[layerIndex].getSource().getTileGrid();
}

4. To display the saved tiles as a map layer, read the tile grid back from the saved JSON, as well as the saved projection and the saved extent. Pass these to OpenLayers to create a generic TileSource layer, along with a file URL template that reflects the directory structure being used for the saved tiles:

function newLayerTileGrid(urlTemplate, tileGridJSON, projCode, displayExtent) {
    var tileGrid = new ol.tilegrid.TileGrid( JSON.parse(tileGridJSON) );
    var source = new ol.source.XYZ({
        url: urlTemplate,
        tileGrid: tileGrid,
        projection: projCode,
    });

    return new ol.layer.Tile({ source: source, extent: displayExtent });
}

Note that the display extent is optional, but it does look a lot nicer to restrict display of the layer to the actual extent that was defined to save, and will avoid unnecessary attempts by OpenLayers to retrieve tiles that do not exist. If the display extent is not used for the layer's extent, then OpenLayers will display whole low-resolution tiles that extend beyond the boundary of the chosen extent where those tiles partially intersected the extent and there are weird anomalies of where these lower zoom level tiles meet up with the full-res tiles for the current zoom level. Also OpenLayers would attempt (and fail) to load tiles for areas of the map that are outside of the extent (not a significant problem in my situation, but good to avoid in any case).

It's also worth noting that in order for JavaScript (inc. OpenLayers) to be able to access saved image tiles the web browser or web API must be configured to allow local file system access. In my case, in Swift, I'm using a WKWebView, and it is loading the root HTML file using the function loadFileURL:allowingReadAccessToURL: in order to allow access to other local files. (Allowing local file access to a web browser running JavaScript is generally a bad idea due to the massive security hole it would create, but for a web view in a non-browser application with limited file access it should be fine.)