0

I've built a mobile mapping device for my Search & Rescue group using a Raspberry Pi (3b+) and its primary purpose is to serve a cache of XYZ tiles via its web server, over its WiFi network. The device is powered using a vehicle's 12V power outlet. Our SAR group uses a number of mapping applications that understand the XYZ tile format. We are regularly outside cell service and satellite internet is problematic, so the goal is to take data with us when we head out on a call.

As a bonus feature I've added the ability to export a georeferenced PDF of any area, with the PDFs content built from the relevant XYZ tiles using GDAL. This works OK but it's slow and I'm wondering if I can remove any of the more expensive steps from the transformation.

The current steps are:

  1. identify all XYZ tiles at a given zoom level that intersect the caller's lat/long bounding box (this is fast)
  2. call GDAL Transform for every XYZ tile, converting it to a .tif with its outputBounds set to the correct bounding box. This can mean several thousand calls to Transform and this part is very slow
  3. call GDAL Warp to merge all Transformed .tif tiles into a single .tif (relying on the newly-added georeferencing data from step 2)
  4. call GDAL Transform to convert the merged .tif to a PDF

I'm already using the Pi's "overlay file system", which I believe writes exclusively to memory / swap space - it's effectively a temporary file system where all changes are lost on reboot - and this made the export significantly faster. This also helps to mitigate the risk of storage corruption as the power supply is regularly interrupted.

It seems like there should be a way to remove step 2, where hundreds or thousands of tiles are individually georeferenced before being merged, but this would require some way of telling GDAL a) the overall bounds of the combined XYZ tiles and b) the order in which the XYZ tiles should be assembled, because without georeferencing we don't know where each tile belongs in the world.

Can anyone suggest a way to make this process more efficient?

8
  • Is your XYZ regular (raster) or irregular (points)? A regular XYZ can be established as a BIL file, taking the origin and cell size from the header to align world coordinates to cells. It sounds like you're projecting on the fly, try to avoid doing that, if you could pin your application down to a UTM zone or use Google Web Mercator for input/output you would save yourself several thousand cycles. Nov 11 '20 at 7:01
  • 2
    The GDAL TMS minidriver that is documented together with the WMS driver gdal.org/drivers/raster/wms.html is almost what you'd need because it knows which x/y/z tiles to read and where they belong without georeferencing them individually. I fear that it is missing the ability to read from a file cache but if you are running http server anyway on your Pi it should be not any problem. For total offline use you could save the tiles into MBTiles or GeoPackage database instead of the directory structure.
    – user30184
    Nov 11 '20 at 7:45
  • 1
    I gave a talk (video) on using GeoServer and MapProxy to do this - GeoServer would allow you to download a PDF directly as a WMS output
    – Ian Turton
    Nov 11 '20 at 10:02
  • @IanTurton thanks but GeoServer and its associated Java baggage is likely far too much for the Pi - I only have 1GB RAM to work with and I want to keep as much as possible free for the temp file system and the device's web server (which is its primary purpose). GeoServer and mapproxy for a "bonus feature" on a low-spec device seems overkill
    – tomfumb
    Nov 11 '20 at 17:37
  • @MichaelStimson thanks for your input. The XYZ tile system conforms to the pseudo-standard system used in Google Maps, OpenLayers, Leaflet etc. - it's already in Web Mercator, but the individual tiles are "georeferenced" by directory and file name index according to the XYZ system's conventions. The georeferencing step (step 2) is not reprojection per se, but creating a duplicate file that has georeferencing metadata attached to it beyond that directory and file name index. I'm just telling the tile where it is so that it can tell GDAL.
    – tomfumb
    Nov 11 '20 at 17:40
1

GDAL can read XYZ tiles from a http server directly with the GDAL TMS minidriver that is documented together with the WMS driver https://gdal.org/drivers/raster/wms.html.

TMS minidriver reads the configuration of the tile service from a small XML file that defines the tiling schema. By the schema the driver knows the georeferencing of the tiles and URLs to access them. Therefore there is no need to georeference the tiles individually.

An example about TMS configuration file:

GDAL_WMS>
    <Service name="TMS">
        <ServerUrl>https://tile.openstreetmap.org/${z}/${x}/${y}.png</ServerUrl>
    </Service>
    <DataWindow>
        <UpperLeftX>-20037508.34</UpperLeftX>
        <UpperLeftY>20037508.34</UpperLeftY>
        <LowerRightX>20037508.34</LowerRightX>
        <LowerRightY>-20037508.34</LowerRightY>
        <TileLevel>18</TileLevel>
        <TileCountX>1</TileCountX>
        <TileCountY>1</TileCountY>
        <YOrigin>top</YOrigin>
    </DataWindow>
    <Projection>EPSG:3857</Projection>
    <BlockSizeX>256</BlockSizeX>
    <BlockSizeY>256</BlockSizeY>
    <BandsCount>3</BandsCount>
    <!--
    <UserAgent>Please add a specific user agent text, to avoid the default one being used, and potentially blocked by OSM servers in case a too big usage of it would be seen</UserAgent>
    -->

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.