I'm working on a project that is creating several hundred thousand GeoTIFFs.
Each GeoTIFF is a projection of species survivability, for the intersection of a climate future and a species. Unlike many of the "lots of GeoTIFFs" problems people have, these are all independent layers that will be viewed one at a time, and each one spans the world... so they cannot be mosaic-ed into a single big layer.
I read on the GeoServer mailing list that having tens of thousands of layers makes GS go slow, so I suspect it will be impractical for me to add all 300,000 layers to the catalog.
For an earlier version of this project, I scripted a web system around MapServer such that it would dynamically grab the source GeoTIFF on first request, then cache it for a while for subsequent tile requests.
Can I do something similar with GeoServer, or should I go back to trying to compile a working recent version of MapServer and MapScript/Python?
Edit: apparently I can use a mosaic with an extra dimension, like "species id", to treat all my GeoTIFFs as a single catalog entry. Will this work for millions of GeoTIFFs?