2

[I've rewritten this question completely in response to comments]

Summary: How to serve GIS data efficiently?

I'm looking to serve raster and vector GIS data as a public service, particularly SRTM1 and SRTM3 data, especially since the https://open-elevation.com/ API appears to be broken (the website is up, but the API times out) and they don't offer SRTM1 regardless.

I'm currently doing this as a test for vector data (timezones from naturalearthdata.com) at:

http://ws.terramapadventure.com:22779/cmd=data&map=timezone&z=0&x=0&y=0

(you can change the values of z, x, and y to anything that would be reasonable for slippy tiles and this should still work)

Because this is JSON data, some browsers may not render it correctly (eg, they might download it instead). Here's a screenshot of how the page looks in my browser:

enter image description here

The screenshot above is limited to the visible part of the page (otherwise it'd be too big), but there is additional data at the bottom of the page:

"elng":"180","x":"0","wlng":"-180","y":"0","map":"timezone","nlat":"90","slat":"-90","z":"0"}

An example using raster data (land use):

http://ws.terramapadventure.com:22779/cmd=data&map=landuse&z=0&x=0&y=0

Screenshot:

enter image description here

And the very similar extra data:

``"elng":"180","x":"0","wlng":"-180","y":"0","map":"landuse","nlat":"90","slat":"-90","z":"0"}`

The vector data comes from this command:

gdal_rasterize ne_10m_time_zones.shp -a zone -ts 512 256 -te -180 -90 180 90 -ot Byte -of Ehdr outfile.bin

with the contents of outfile.bin stuffed into the data field of a JSON object (it's pure binary data so it looks really ugly-- I would switch it to base64, but I'm trying to get away from creating my own protocol entirely).

The raster data comes from this command:

gdal_translate landuse.bin -outsize 512 256 -projwin -180 90 180 -90 -ot Byte -of Ehdr outfile.bin

where landuse.bin is in Ehdr format, but mounted on a squashfs drive to save space.

In both cases, the z, x, and y values determine the latitude-longitude rectangle for which the end user wants results, similar to OSM slippy tiles, but I'm using the Plate Carree, so the values are a little different.

As per the gdal commands above, I then split the rectangle into 512 equally spaced longitudes and 256 equally spaced latitudes, and return the data for those latitudes and longitudes.

Since the grid is 512 x 256, that's 131,072 values total. In the two examples above, there are fewer than 64 values for both timezones and land use types, so each value can be fit into 1 byte (and actually into 6 bits, which might be useful later to minimize bandwidth), so I send 131,072 bytes. In other cases, such as elevation, which requires 2 bytes per value (actually 14 bits), I would send a multiple of 131,072 bytes (eg, 262,144 bytes).

I'm ok with keeping Plate Carree tiles, but I'd like to serve the data itself in "proper" GIS format, for example WFS or WCS, or some other format that GIS users already use. My issues:

  • I want to serve the data relatively quickly. For example, when I tried storing the raster data as GeoTIFFs, the gdal_translate command was too slow (gdalwarp is even slower). By storing rasters as Ehdr files, data retrieval is much faster (which makes sense, since Ehdr is pretty much just a string of bytes). The Ehdr files are much larger than GeoTIFFs (squashfs compression helps), but I couldn't find a smaller format that worked as quickly.

  • I'd prefer to use an existing tool (such as GeoServer, but not necessarily GeoServer specifically) instead of writing my own code, which is what I'm doing now (https://github.com/barrycarter/bcapps/tree/master/MAPS/bc-mapserver.pl) since I'm sure existing code is better tested.

  • I'm currently using Perl, but this may not be an issue, since I'm hoping to transition to an existing product (in any language).

  • I want to provide the data in a format that's easily accessible. Sending a stream of raw bytes is probably not ideal, but I'm concerned sending something like a whole GeoTIFF would take too long for the user to parse. Although people can use the server for anything, I'm anticipating mapping usage, so the data I send should be easily parseable.

  • I'm not sure GeoServer can do what I want: it has WCS and WFS, but I'm not sure it can rasterize/resample on the fly.

  • Since the Ehdr files are fairly large and the gdal commands are fairly fast, I'd like to avoid tile caching. I'm also worried tile caching would enforce a style on the end user and I want to avoid that: I want to send raw data and let users decide if and how to "color" it.

  • I believe raster data is more useful to end users, so I'm not interested in providing SHP files or other vector data as an end product.

  • To minimize bandwidth, I'd like to send the data in as compressed a format as possible, but without making it too time-consuming to uncompress on the user's side.

  • Your example link is giving an error message, so I'm not sure what your expected output looks like. On the one hand, you're saying you want to send raw data, but you're also saying you want to send a rendered shapefile, which is not really raw data anymore. Could you include an example of what you want in the question body? – PhilippNagel Jun 5 at 17:23
  • The link should work (but you have to use the whole link, not just the host and port). For vector files, I'm looking to sample data at 512 by 256 points in the requested area and return those 131,072 chunks of data. Some of the data are 2 or even 4 byte, so I may return a multiple of that number of bytes. I'll also edit the question. – barrycarter Jun 5 at 17:47
  • I was able to view the link in Firefox, but it fails in Chrome. Anyway, if I understand correctly, you don't want to send the data in a vector format, right? A common format for that would be GeoJSON. How is this output going to be consumed? An application you are developing, or will this be an API to be used by others directly? I think a good approach would be doing this in Python with the Python GDAL bindings, and something like Flask to serve the data.To just get binary rasters from a shapefile, Geoserver WMS (instead of WFS) would also work well. – PhilippNagel Jun 5 at 20:34
  • @PhilippNagel Your comments were very helpful, thank you! I've now rewritten the question to hopefully be more detailed. – barrycarter Jun 6 at 15:08
  • WCS and WFS are not formats, rather service interface standards. WCS unlike WFS defines functions for scaling the data and creating new images/outputs on the fly. If GeoServer doesn't do it, look at Rasdaman – nmtoken Jun 6 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.