Should GDAL be set to produce GeoTIFF files with compression? Which algorithm should be used?

Question

I have a folder of GIS data that consists mainly of GeoTIFF files. The whole set weighs in at about 1.2 GB. I noticed that if I pack the contents into a tarball, it smashes down to about 82 MB. I would like to check the set into a revision control system sot it can be worked on by other people and it looks like there is some space that can be squeezed out.

The GDAL GeoTIFF driver page lists plenty of options that may be used to create compressed GeoTIFF files. There are also plenty of options that affect the way each algorithm works.

The help page does a good job at describing the options but doesn't elaborate on how to select an algorithm or the tradeoffs that are associated with the varying level of compression. This leads to the following questions:

• The pros of using compression are a dramatic savings in space. What are the cons? Is information lost when the image is compressed?

• How should one go about choosing an algorithm and compression level. Do some types of images lend themselves to a certain algorithm?

Answer 1

To select compression method you need to use a command like:

gdal_translate -co "COMPRESS=method" src_dataset dst_dataset

When you use compression biggest trade-off is extra processing time which is required to uncompress the image, and after uncompressing the image would still consume same amount of memory. About information loss there are two basic types of compression:

• lossless - which preserve original data values
• lossy - which degrade data to save even more space

You would lossless algorithms when original data values must be preserved, like DEMs, or raster features. Algorithms like PACKBITS, DEFLATE and LZW are lossless and can be ordered according compression ratio:

1. LZW - highest compression ratio, highest processing power
2. DEFLATE
3. PACKBITS - lowest compression ratio, lowest processing power

Compression ratio still depends on data, if the data has a lot of similar values PACKBITS will yield good results.

Contrary to lossless you would use lossy algorithms like JPEG to compress rasters that don't have to return exact values. For instance, orthophotos or satellite imagery can be compressed using lossy algorithms.

Answer 2

With lzw and deflate compression using -co predictor=2 can help with imagery that is smoothly varying as it compresses the differences from pixel to pixel instead of the absolute values, and these will tend to be small and have more patterns (ref). Predictor is only useful with lzw and deflate compression, the option has no effect with other methods.

gdal_translate -co compress=lzw -co predictor=2 ...

The predictor savings can be dramatic. I just re-compressed a directory of 16bit geotiff elevation models using up 17GB with the default LZW settings into just 5GB with predictor=2.

There is conflicting info on the differences between predictors 2 & 3 and when each is best applied (ref1, ref2). Perhaps fuel for another question.

Another easy option for savings is -co tiled=yes. There are some software which can't read tiled images, but those are becoming rarer and mostly outside of GIS (I don't know of any main stream GIS software now that doesn't read them).

To build on @alfonx's answer of using compressed overviews: This allows the base image to be stored lossless, for data integrity, and the pyramids to be lossy, for speed and some space savings. It's almost the best of both worlds. For the smallest possible overviews with gdaladdo on RGB images: use jpeg compression, averaged or gaussian resampling instead of the default nearest neighbour (makes the overviews smoother), and YCBCR photometric overview. See the gdaladdo reference page for more info on these options (though it doesn't say much about what photometric is all about).

This is part of a windows batch file I use to apply external jpeg overviews to all tiffs in a directory:

set _opts= -r gauss --config PHOTOMETRIC_OVERVIEW YCBCR ^
--config COMPRESS_OVERVIEW JPEG --config JPEG_QUALITY_OVERVIEW 85

for %%a in (*.tif) do gdaladdo -ro %_opts% %%a 2 4 8 16 32 64

Notes

GDAL 1.6.0 introduced gauss resampling which can lead to better results average in case of sharp edges with high contrast or noisy patterns. Powers of 2 levels (2 4 8 ...) should be used so a 3x3 resampling Gaussian kernel is selected.

JPEG_QUALITY_OVERVIEW 85 - if not specified the default of 75% is used, which does yield smaller file, but I find 85% a better compromise in the size vs quality trade off.

Update, 2015: GDAL 1.8 and 2.0 have introduced a lot of new options not covered here and which I haven't had time to digest. Read the official gtiff format page, I'm sure there are additional useful settings detailed.

Answer 3

For big rasters GeoTiff offers the possibility to store (pre-)downscaled overviews as extra images to the GeoTiff file. This can be done with gdaladdo (= GDAL ADD Overview). When creating these overviews, you can manually tell gdal to compress them too:

gdaladdo --config COMPRESS_OVERVIEW JPEG 

Speeds up viewing your data without adding too much size. Note: Geotools applications like Geoserver, uDig, AtlasStyler, Geopublisher can all use this feature and profit from overviews.

Answer 4

For those using newer GDAL versions, there's also the lossless ZStandard (ZSTD) compression (GDAL>=2.3) and lossy Limited Error Raster Compression (LERC) compression (GDAL>=2.4) choices available.

Generally speaking though, ZSTD offers faster data read speeds than both LZW and DEFLATE with similar compression ratios, though it can be somewhat slower when writing the file (depending on what settings you use).

If you're not that fussed about data precision (e.g. only doing visualization rather than analysis), then LERC might be a good option. There is a MAX_Z_ERROR setting that allows you to tweak how much precision you are willing to sacrifice. E.g. a MAX_Z_ERROR=0.001 or 1mm gave a space saving of 50% in one benchmark (see ref).

The best part is that you can also combine LERC with ZSTD using COMPRESS=LERC_ZSTD! Or if you prefer using DEFLATE, you can do COMPRESS=LERC_DEFLATE. See also full list of combinations/settings at the official GDAL GeoTIFF docs https://gdal.org/drivers/raster/gtiff.html#creation-options

More details and full benchmark comparisons can be found at this valuable reference:

• 'Guide to GeoTIFF compression and optimization with GDAL' by @kokoalberti

Answer 5

To enable partial image decompression, simply use TILED=YES .

Peter

Answer 6

The answers by @dodobas and @matt-wilkie cover most everything relating to the acts of compressing and blurring with GDAL to reduce image size.

I would like to add two things:

• the file-format documentation from GDAL: http://www.gdal.org/frmt_gtiff.html;
  • See the creation options (-co), specifically:
    • COMPRESS
    • NUM_THREADS
    • PREDICTOR
    • ZLEVEL
• and that it is essential to verify that the software that will be consuming the GeoTIFFs:
  • supports the desired compression method;
  • recommends using compression.

[Edit 2021-06-24]

Thanks goes to @bugmenot123 for pointing out that GeoServer has changed their recommendation to say:

Generally speaking, if I/O is the bottleneck, compression can help a lot as it reduces the cost of I/O although at the expenses of some CPU cycles.

If important, then performance testing should be done with varying compression methods and take into account the use of overviews, tiling, and storage medium (consumer or budget disk verse enterprise grade disk or SSD).

Answer 7

Ultimately you'll probably need to experiment with the different options and see what meets your needs.

I've been making increased use of JPEG-compressed GeoTIFFs over wavelet-based formats. My results have been pretty good. Using GDAL to do this has yielded compression ratios comparable to wavelet-based formats without too much data loss. The performance hit that comes with decompression has been acceptable.

What I like most about this approach is that GeoTIFF support is almost universal, while support for wavelet-based formats isn't always assured and is sometimes subject to thorny licensing issues.

Answer 8

My experience comparing GeoTIFF vs. Earth Resource Mapping's ECW (Enhanced Compressed Wavelet) compression is that ECW is orders of magnitude better when compressing high resolution aerial photos. Another important advantage of wavelet based compression is that, unlike older formats as GeoTIFF, JPEG - not JPEG 2000 -, just a portion of the image can be decompressed [ref. 1]. The importance of this advantage must not be underestimated, specially when working with "larger than about half computer memory size.".

It seems - I have never had the chance of testing it - that MrSID, another propietary, wavelet based file format, also exhibits higher compression ratios than "older" formats and selective decompression.

ref. 1: http://www.ifp.uni-stuttgart.de/publications/phowo01/Ueffing.pdf