[I've rewritten this question completely in response to comments]
Summary: How to serve GIS data efficiently?
I'm looking to serve raster and vector GIS data as a public service, particularly SRTM1 and SRTM3 data, especially since the https://open-elevation.com/ API appears to be broken (the website is up, but the API times out) and they don't offer SRTM1 regardless.
I'm currently doing this as a test for vector data (timezones from naturalearthdata.com) at:
(you can change the values of z, x, and y to anything that would be reasonable for slippy tiles and this should still work)
Because this is JSON data, some browsers may not render it correctly (eg, they might download it instead). Here's a screenshot of how the page looks in my browser:
The screenshot above is limited to the visible part of the page (otherwise it'd be too big), but there is additional data at the bottom of the page:
An example using raster data (land use):
And the very similar extra data:
The vector data comes from this command:
gdal_rasterize ne_10m_time_zones.shp -a zone -ts 512 256 -te -180 -90 180 90 -ot Byte -of Ehdr outfile.bin
with the contents of outfile.bin stuffed into the data field of a JSON object (it's pure binary data so it looks really ugly-- I would switch it to base64, but I'm trying to get away from creating my own protocol entirely).
The raster data comes from this command:
gdal_translate landuse.bin -outsize 512 256 -projwin -180 90 180 -90 -ot Byte -of Ehdr outfile.bin
where landuse.bin is in Ehdr format, but mounted on a squashfs drive to save space.
In both cases, the z, x, and y values determine the latitude-longitude rectangle for which the end user wants results, similar to OSM slippy tiles, but I'm using the Plate Carree, so the values are a little different.
As per the gdal commands above, I then split the rectangle into 512 equally spaced longitudes and 256 equally spaced latitudes, and return the data for those latitudes and longitudes.
Since the grid is 512 x 256, that's 131,072 values total. In the two examples above, there are fewer than 64 values for both timezones and land use types, so each value can be fit into 1 byte (and actually into 6 bits, which might be useful later to minimize bandwidth), so I send 131,072 bytes. In other cases, such as elevation, which requires 2 bytes per value (actually 14 bits), I would send a multiple of 131,072 bytes (eg, 262,144 bytes).
I'm ok with keeping Plate Carree tiles, but I'd like to serve the data itself in "proper" GIS format, for example WFS or WCS, or some other format that GIS users already use. My issues:
I want to serve the data relatively quickly. For example, when I tried storing the raster data as GeoTIFFs, the
gdal_translatecommand was too slow (
gdalwarpis even slower). By storing rasters as Ehdr files, data retrieval is much faster (which makes sense, since Ehdr is pretty much just a string of bytes). The Ehdr files are much larger than GeoTIFFs (squashfs compression helps), but I couldn't find a smaller format that worked as quickly.
I'd prefer to use an existing tool (such as GeoServer, but not necessarily GeoServer specifically) instead of writing my own code, which is what I'm doing now (https://github.com/barrycarter/bcapps/tree/master/MAPS/bc-mapserver.pl) since I'm sure existing code is better tested.
I'm currently using Perl, but this may not be an issue, since I'm hoping to transition to an existing product (in any language).
I want to provide the data in a format that's easily accessible. Sending a stream of raw bytes is probably not ideal, but I'm concerned sending something like a whole GeoTIFF would take too long for the user to parse. Although people can use the server for anything, I'm anticipating mapping usage, so the data I send should be easily parseable.
I'm not sure GeoServer can do what I want: it has WCS and WFS, but I'm not sure it can rasterize/resample on the fly.
Since the Ehdr files are fairly large and the gdal commands are fairly fast, I'd like to avoid tile caching. I'm also worried tile caching would enforce a style on the end user and I want to avoid that: I want to send raw data and let users decide if and how to "color" it.
I believe raster data is more useful to end users, so I'm not interested in providing SHP files or other vector data as an end product.
To minimize bandwidth, I'd like to send the data in as compressed a format as possible, but without making it too time-consuming to uncompress on the user's side.