4

I have a 500MB GeoTIFF with 3 bands on disk. I can easily read a very small, 25x25-pixel subwindow using GDAL like so:

gdal_translate -srcwin 600 550 25 25 hugeimage.tif smallwindow.tif

Does this operation have to read the entire large GeoTIFF into memory, or is it able to seek to the appropriate bytes on disk and read only them?

Does the answer change for selecting individual bands, instead of clipping pixels?

Does the answer change if the GeoTIFF is compressed?

2

File-oriented operations rarely "read the whole image into memory", however:

  1. It will always need to read the full header (which varies in size by format)
  2. It will always need to allocate a row buffer (or tile buffer, if the format is tile-oriented) for input and another for output (in the case of conversion).
  3. If a compression algorithm is used, additional memory will be needed to hold the decompressed copy
  4. The application has no control over the operating system, which may cache everything it's asked to read until some threshold is reached.
  5. Likewise, most disks have a RAM cache, and the OS has no control over what the disk retains.

Generally speaking, no competently-developed application (a class of which GDAL is certainly a member) will ever do more I/O than is necessary. Such applications will also strive to minimize memory use, but if the choice is between using an extra 64Kb of RAM or re-reading a file block (and especially re-reading a file block repeatedly), then using memory is going to win out. Back in the day of systems with 8-32Mb RAM, this was a cruel choice, but with 8Gb-256Gb RAM, a 64K block is a drop in the bucket.

In your particular case, the I/O required to read a 25x25 pixel window depends on:

  • The pixel depth of the image (unknown)
  • The pixel width of the image (unknown)
  • The number of bands (3)
  • The pixel interleave of the output image (assumed none)
  • The tile blocking of the image (unknown)
  • Any pyramiding that may be embedded in the format (assumed none)
  • Any compression done per tile (assumed none)

If we assume that it is 8-bit depth, a square image, and no tile blocking, then then we have a 13219x13219 image with a 62117 byte header. The application would need to read the header, plus 25 rows times three bands. The input I/O would be 101774 bytes, but the RAM used would only be 75336 (header plus the row buffer). On output, assuming a 1024 byte header, only 2899 bytes would be written, requiring only the header (1024) and the row buffer (25) in RAM.

If the same image was instead 13056 (W) x 13384 (H), with a 256x256 tiling scheme (and no compression), at most 4 tiles would need to be read. If we assume best case, then the I/O is 260096 bytes (63488 header plus 256x256 tile buffer, three times), and the memory is 129024 bytes (header plus one tile). But if you were unlucky, and the 25x25 pixel chunk you wanted crossed four tiles, then you'd need 849920 bytes of input I/O, and 194560 bytes of RAM (because a second tile would need to be cached to fill the output buffer, vice reading an extra tile 24 times for each band).

So, we see that the details of how much I/O and RAM is required for any specific file is critically linked to the actual file organization and the intended use (neither of which may be optimal with respect to the other), but it's usually safe to assume that GDAL will use the minimum necessary for the task (unless otherwise influenced by unwise parameters).

5

It doesn't have to read the whole thing. GDAL is heavily geared to sensible access either line by line or block by block (for tiles). It won't matter if you are selecting bands, and no the answer won't change if it's compressed - but the real details will depend on which compression and which internal tiling is used. See "Reading Raster Data" here

http://www.gdal.org/gdal_tutorial.html

As it says,

There are a few ways to read raster data, but the most common is via the GDALRasterBand::RasterIO() method. This method will automatically take care of data type conversion, up/down sampling and windowing.

The definition for GDALRasterBand::RasterIO() shows the crux of it, where this key method accepts a query for offset, size of window read, and size of the output. The way the file is structured internally will matter - say if it's tiled, since it will depend on whether your query strides multiple tiles or exists in one.

http://www.gdal.org/classGDALRasterBand.html#a30786c81246455321e96d73047b8edf1

I think you'll find GDAL does this with very minimal computation and I/O.

  • 2
    I think the key point is that if you have some kind of compression on the large image, and there is no tiling / striping, then its going to be much worse than if there is sensible tiling. – BradHards Dec 5 '15 at 10:24
  • Good point @BradHards. Tiling seems to be key to my answer; I hadn't realized compression is done per-tile. – Jeff G Dec 5 '15 at 22:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.