I have a 60Gb raster vegetation dataset. I need to be able to access single pixels relatively quickly. At the moment, I am retiling at a fixed size, and saving those, indexing them in Python, and accessing them from there.
However, this results in tiles with a wide array of sizes - places with consistent vegetation get compressed well, and are really small, places with a lot of variance are much bigger. I would ideally like to optimise the file sizes so that my index is not too huge (~10K files maybe), but so that file access is also minimised - for any given pixel, I would rather not try and load a 100Mb file.
Is there an easy method of retiling in one step that would tile a dataset into (very) approximately uniform file sizes? I guess I could tile at a large size, and then re-tile each of the largest files into smaller files, and repeat. But it would be nice if there was a way that didn't require multiple steps.