I'm working on a project that is creating several hundred thousand GeoTIFFs.

Each GeoTIFF is a projection of species survivability, for the intersection of a climate future and a species. Unlike many of the "lots of GeoTIFFs" problems people have, these are all independent layers that will be viewed one at a time, and each one spans the world... so they cannot be mosaic-ed into a single big layer.

I read on the GeoServer mailing list that having tens of thousands of layers makes GS go slow, so I suspect it will be impractical for me to add all 300,000 layers to the catalog.

For an earlier version of this project, I scripted a web system around MapServer such that it would dynamically grab the source GeoTIFF on first request, then cache it for a while for subsequent tile requests.

Can I do something similar with GeoServer, or should I go back to trying to compile a working recent version of MapServer and MapScript/Python?

Edit: apparently I can use a mosaic with an extra dimension, like "species id", to treat all my GeoTIFFs as a single catalog entry. Will this work for millions of GeoTIFFs?

  • You do not likely need so many layers but you can use one layer as an image mosaic docs.geoserver.org/latest/en/user/data/raster/imagemosaic/…. With MapServer I would have used tile index layer mapserver.org/optimization/tileindex.html. – user30184 Aug 15 '17 at 6:43
  • Thanks user30184 but my tiffs aren't little parts of a big layer, they're all world-sized and need to be viewable independently of each other (I updated the question to be clearer about that) – Daniel Baird Aug 15 '17 at 7:14
  • It does not matter. There can be holes in the mosaic but when user zooms to desired area the correct image shows. It does not require much programming to store the bounding boxes of individual sites and offer shortcut for your users to find the location. If images overlap you can use view parameters for making a SQL like query to the index. – user30184 Aug 15 '17 at 7:21
  • 1
    Most common use case is to use time as filtering parameter docs.geoserver.org/latest/en/user/tutorials/… but other parametes can be used as well but maybe not out-of-the-box. – user30184 Aug 15 '17 at 7:29
  • 1
    You do understand mosaics but from too narrow point of view. Mosaic is an index that can be used to organize tiles side by side but there are more dimensions in the world. This OpenLayers demo is using some WMS that has a time-enabled mosaic, or maybe better word, index dev.openlayers.org/examples/wmst.html. Imagine that mosaic is a database. While the usual query is select image from index where ST_Intersects(footprint,BBOX) you can as well make the system to pick the right image for you with select image from index where survivability_factor_1=xx and survivability_factor_2=yy. – user30184 Aug 25 '17 at 6:46

Yes, GeoServer can handle hundreds of thousands of GeoTiffs and it will be very fast when only a few of them are loaded on each request.

This is possible using a single Imagemosaic with a PostGIS database, that has the species ID included. See this solution based on CQL Filters. For good performance, create an Index on the species ID field in your database.

That article is a bit dated. It's possible to include custom dimensions already (since 2.11.2)

While custom dimensions will work, it will also blow up your GetCapabilities document, because it will include a list of all possible values, so the CQL method might be the way to go.

Make sure to limit the number of tiles per Request (MaxAllowedTiles).

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.