1

I have a 60Gb raster vegetation dataset. I need to be able to access single pixels relatively quickly. At the moment, I am retiling at a fixed size, and saving those, indexing them in Python, and accessing them from there.

However, this results in tiles with a wide array of sizes - places with consistent vegetation get compressed well, and are really small, places with a lot of variance are much bigger. I would ideally like to optimise the file sizes so that my index is not too huge (~10K files maybe), but so that file access is also minimised - for any given pixel, I would rather not try and load a 100Mb file.

Is there an easy method of retiling in one step that would tile a dataset into (very) approximately uniform file sizes? I guess I could tile at a large size, and then re-tile each of the largest files into smaller files, and repeat. But it would be nice if there was a way that didn't require multiple steps.

  • Try tiling without compression as striped GeoTIFF, this will give a fairly uniform tile size and you will not be wasting CPU cycles to uncompress the data when reading (or compressing when writing). That said though if you provide more information about how you are accessing the pixels in GDAL we could give some more specific advice; personally I have not found that reading a chunk from a very large (uncompressed) image is slower than reading a whole tile. If performance is a concern accessing a raster with the C++ GDAL API is much faster than doing the same in python, C# is in between both. – Michael Stimson May 17 at 1:37
  • 1
    The first rule of optimization is to measure performance across all your modules and start optimizing the most costly component. I have my doubts that this is a major issue for you. – Vince May 17 at 1:46
  • @MichaelStimson: total file size and disk IO is far more of a problem than CPU usage for decompression. This is one dataset of many, with each of which we a similar problem (probably 2Tb total, with LZW compression when we are done). But yes, Vince is right, we should do more performance testing to pin point problems. It's on the cards. – naught101 May 17 at 3:21
  • 1
    Space is cheap, time is expensive. The problem with compressed images is that there is no fixed length for a given scanline (row) so all the preceding lines need to be evaluated first, a block of data read and then decompressed into a scanline before the desired section can be loaded into the buffer provided to the raster io. You can improve the access by converting to a tiled TIFF and reading a tile gdal.org/… from the raster as the tiles are indexed so compression or not matters little. – Michael Stimson May 17 at 3:44
  • 1
    What about treating all data as a single coverage, and accessing the data required through WCS with scaling and subletting as appropriate, using say Rasdaman (rasdaman.org)? – nmtoken May 17 at 8:31
2

Raster tiles tend to use a regular grid and each tile on certain zoom level covers as large area. If data is dense then the tile will be bigger as compressed. As uncompressed you would get tiles with uniform size but they would be all big. I fear you are out of luck with current slippy map tiling systems.

You are after something like r-tree https://en.wikipedia.org/wiki/R-tree. You can find such a feature in Cesium 3D tiles https://cesium.com/blog/2015/08/10/introducing-3d-tiles/. Have a look at image "An adaptive quadtree-like subdivision based on the distribution of buildings in Cambridge".

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.