Database for offline slippy map tiles [closed]

Question

I currently have an offline HTML5 map application (built on Leaflet & KendoUI with custom additions) that has an app manifest and works fine on multiple platforms. However, I am hesitant to use the manifest to store the actual map tiles this way(PNG files stored as a TMS style Tile Cache).

Issues:

• There could be a lot of tiles (10MB - 50MB) in about 1,000 PNG files
• The initial download could be really slow (and hard to show progress to the user)
• App Manifests work or they don't if they don't the entire offline caching will fail (accord to [whatwg.org][1])
• The offline user will occasionally reconnect and need to get refreshes of the Tiles, These are small deltas but the app manifest mechanism would reload all the js, css, and PNG files as soon at the manifest updates

Alternative idea: keep the web application separate from the storage of the slippy map tiles. Store the tiles in a web application friendly database

Update:

[PouchDB recently added support for binary blobs. I am getting good initial results. See: https://stackoverflow.com/questions/16721312/using-pouchdb-as-an-offline-raster-map-cache ]

• This is suggested by Ben Nolan http://bennolan.com/2011/06/03/offline-mapping.html
• Similar work on Maps on a Stick: http://developmentseed.org/blog/2010/oct/02/maps-stick-version-2-released/ ([deprecated][2])
• MBtiles http://mapbox.com/developers/mbtiles/
• TileStream https://github.com/mapbox/tilestream
• Lous Remi: http://louisremi.com/2011/10/07/offline-web-applications-were-not-there-yet/

Question: What does the collective wisdom (and experience) say about the following choices for a JavaScript friendly DB:

1. SqlLite

• Seems like you need to create a native app wrapper for this to allow it to talk to JavaScript
• E.g. Add you DLL to a native program for windows, and PhoneGap for android/IOS

2. WebSQL

• depricated
• but it was a SQL Lite that I could easily generate and distribute from the host web server

3. IndexDB

• Storing blobs seems to work see: https://hacks.mozilla.org/2012/02/storing-images-and-files-in-indexeddb/
• I am concerned if this is the only way to initially populate the DB
• Is this basically a SQLLite file? Can I ship it for bulk DB upload?
• I am leaning towards this as a solution. Are their gotchas I don't know about?

Requirements:

• Fast initial population (via download) to the client Web DB
• Compatible with current Leaflet TileLayer API (i.e. I would rather not write a custom layer, but if needed ...) (e.g. MbTiles)
• Platform: Windows Laptops, but Android and IOS tablets desired (I can wait till IndexDB is released, don't need immediate support)
• I would rather not write a native app (EXE, IOS, Android) but it if that is the best way...
• Server Side generation of web maps (this will be an automated process). The user picks a location, chooses maps, and they get dynamically transformed and turned into a slippy tile cache (this work is already largely done).
• Fast bulk initial download
• Map change delta refreshes (I will write this logic, based on constant stock numbers, and update date logic)
• Minimal impact on current Leaflet & KendoUI web application

Key background idea: while the web app is fairly stable, the slippy map tiles are generated on the fly for your location and what type of problem you are doing (at the back-end). So I thought of two other ways for transferring the initial 'big bang' and then updates:

Zip File (probably not a good idea - as it adds server load) also expansion on client machine will require user interaction, but it does allow slippy tiles to use local url's

HTML5 File API: I have not looked at this in great detail. But it appears has most operations to create a local file tree in TMS format: http://www.html5rocks.com/en/tutorials/file/filesystem/ what will be interesting to test is the performance (e.g. can I use web works to maximize the bandwidth to disk and across the net). IndexDB is not widely implemented to be web worker friendly (sync interface: https://stackoverflow.com/questions/10698728/indexeddb-in-web-worker-on-firefox

I have found some additional info about using IndexDB with Leaflet:

https://github.com/calvinmetcalf/leaflet.pouch (syncs couchdb with indexdb for offline) Also here are some tests for the read/write speeds for indexdb, websql, & local store: http://jsperf.com/indexeddb-vs-localstorage/15

And here is how to use the read/write file API from javascript: (and also to ask to increase storage limits) http://www.html5rocks.com/en/tutorials/file/filesystem/

---

Tom MacWright (aka tmcw) gave some good feedback. Your example is really going to help when I get to creating custom layers to ingest the binary blobs.

I did some testing yesterday with IndexedDB and by using some polyfills and libraries I think it will solve my issues. Now it is time to put some sweat equity into this, and I will report back.

---

If you want to see my results of my study on client side databases see:

https://stackoverflow.com/questions/14113278/storing-image-data-for-offline-web-application-client-side-storage-database

---

Answer 1

PhoneGap and MBTiles.

WebSQL & IndexDB will not be enough. 'Windows Laptops' will not be the same code as mobile devices.

Answer 2

Results Offline blob cache for PNG slippy maps

Testing

• 171 PNG files (total of 3.2MB)
• Platforms tested: Chrome v24, FireFox 18, IE 10
• Should also work with Chrome & FF for Android

Fetch from web server

• using XHR2 (supported on almost all browsers) for blob download from web server
• I went with XHR2-Lib by Phil Parsons, which is very much like JQUERY .ajax()
  • https://github.com/p-m-p/xhr2-lib

Storage

• IndexedDB for IE and FireFox
• Chrome: Polyfill (blob stored using FileSystem API, reference kept in IndexedDB) polyfill
• A Must read article on "How the browsers store IndexedDB data"
  • http://www.aaron-powell.com/web/indexeddb-storage
• Note: FireFox uses SQLlite for the NOSQL IndexedDB. That might be the reason for the slow performance. (blobs stored separately)
• Note: Microsoft IE uses the extensible storage engine:
  • http://en.wikipedia.org/wiki/Extensible_Storage_Engine
• Note: Chrome uses LevelDB http://code.google.com/p/leveldb/

Display

• I am using Leaflet http://leafletjs.com/ to show the map tiles
• I used the functional tile layer plugin by Ishmael Smyrnow for fetching the tile layer from the DB
  • https://github.com/ismyrnow/Leaflet.functionaltilelayer
• I compared the DB-based tiles layer with a purely local (localhost://) storage
• There is no noticeable difference in performance! between using IndexedDB and local files!

Results

• Chrome: Fetch (6.551s), Store (8.247s)
• FireFox: Fetch (0.422s), Store (34.519s)
• IE 10: Fetch (0.668s), Store: (0.896s)

Answer 3

so you almost definitely don't want to use leaf.pouch, I made that with vector data in mind not tiles, you'd probably be better off using strait up PouchDB to store your tiles, as you can also replicate from CouchDB for the fast initial load, The answer by @tmcw above would also work if you have more of an phone app as opposed to a mobile web page.

Answer 4

use the standard SQLite3 container MBTILES format with tiles table with tile_data blob field with PNG or JPG or WebP or GZipped PBF integrated it with MBTILES.JS https://github.com/tilemapjp/mbtiles.js/tree/master https://www.npmjs.com/package/Leaflet.TileLayer.MBTiles

you can convert TMS or XYZ Tiles into mbtiles with MBUTIL by MapBox. if you need to generate folder of tiles first use GDAL2TILES_Parallel.py or gdal2tilesp.py the parallel multiprocessing python implementations of the orginal. They both require GDAL.

Answer 5

it will read local and remote mbtiles. Package as mobile app with IONIC or React Native. or Desktop with Electron Atom. MBTILES are the way to go