49

I have a folder of GIS data that consists mainly of GeoTIFF files. The whole set weighs in at about 1.2 GB. I noticed that if I pack the contents into a tarball, it smashes down to about 82 MB. I would like to check the set into a revision control system sot it can be worked on by other people and it looks like there is some space that can be squeezed out.

The GDAL GeoTIFF driver page lists plenty of options that may be used to create compressed GeoTIFF files. There are also plenty of options that affect the way each algorithm works.

The help page does a good job at describing the options but doesn't elaborate on how to select an algorithm or the tradeoffs that are associated with the varying level of compression. This leads to the following questions:

  • The pros of using compression are a dramatic savings in space. What are the cons? Is information lost when the image is compressed?

  • How should one go about choosing an algorithm and compression level. Do some types of images lend themselves to a certain algorithm?

80

To select compression method you need to use a command like:

gdal_translate -co "COMPRESS=method" src_dataset dst_dataset

When you use compression biggest trade-off is extra processing time which is required to uncompress the image, and after uncompressing the image would still consume same amount of memory. About information loss there are two basic types of compression:

  • lossless - which preserve original data values
  • lossy - which degrade data to save even more space

You would lossless algorithms when original data values must be preserved, like DEMs, or raster features. Algorithms like PACKBITS, DEFLATE and LZW are lossless and can be ordered according compression ratio:

  1. LZW - highest compression ratio, highest processing power
  2. DEFLATE
  3. PACKBITS - lowest compression ratio, lowest processing power

Compression ratio still depends on data, if the data has a lot of similar values PACKBITS will yield good results.

Contrary to lossless you would use lossy algorithms like JPEG to compress rasters that don't have to return exact values. For instance, orthophotos or satellite imagery can be compressed using lossy algorithms.

  • 5
    +1 for the nice answer. PACKBITS is a form of run-length encoding (en.wikipedia.org/wiki/Run-length_encoding) which will work well for data with lots of adjacent same values (if for example, you have lots of NULLs or a classified raster) and LZW is a more robust algorithm which is effective on more kinds of data. The general trade-off is between space and speed as mentioned, so what's appropriate depends on your use and data. Also, some software doesn't support certain kinds of GeoTiff compression. – scw Aug 12 '10 at 19:04
  • 3
    this is a good, relevant post linfiniti.com/2011/05/… – oeon Jun 1 '11 at 5:01
  • 1
    Good answer, it summarizes your options well. Remember also that each of those compression methods has parameters you can set, which will influence the outcome considerably. @j03lar50n, glad you found my blog article useful ... – R Thiede Sep 1 '11 at 20:34
  • beautiful answer! so simple and right to the point. – sys49152 Jan 31 '18 at 13:41
  • @scw could you say more about what software doesn't support certain types of compression - specifically, is there any software that won't support lzw or packbits? Or are you mostly referring to less common algorithms? – David LeBauer Jan 31 at 21:27
28

With lzw and deflate compression using -co predictor=2 can help with imagery that is smoothly varying as it compresses the differences from pixel to pixel instead of the absolute values, and these will tend to be small and have more patterns (ref). Predictor is only useful with lzw and deflate compression, the option has no effect with other methods.

gdal_translate -co compress=lzw -co predictor=2 ...

The predictor savings can be dramatic. I just re-compressed a directory of 16bit geotiff elevation models using up 17GB with the default LZW settings into just 5GB with predictor=2.

There is conflicting info on the differences between predictors 2 & 3 and when each is best applied (ref1, ref2). Perhaps fuel for another question.

Another easy option for savings is -co tiled=yes. There are some software which can't read tiled images, but those are becoming rarer and mostly outside of GIS (I don't know of any main stream GIS software now that doesn't read them).

To build on @alfonx's answer of using compressed overviews: This allows the base image to be stored lossless, for data integrity, and the pyramids to be lossy, for speed and some space savings. It's almost the best of both worlds. For the smallest possible overviews with gdaladdo on RGB images: use jpeg compression, averaged or gaussian resampling instead of the default nearest neighbour (makes the overviews smoother), and YCBCR photometric overview. See the gdaladdo reference page for more info on these options (though it doesn't say much about what photometric is all about).

This is part of a windows batch file I use to apply external jpeg overviews to all tiffs in a directory:

set _opts= -r gauss --config PHOTOMETRIC_OVERVIEW YCBCR ^
--config COMPRESS_OVERVIEW JPEG --config JPEG_QUALITY_OVERVIEW 85

for %%a in (*.tif) do gdaladdo -ro %_opts% %%a 2 4 8 16 32 64

Notes

GDAL 1.6.0 introduced gauss resampling which can lead to better results average in case of sharp edges with high contrast or noisy patterns. Powers of 2 levels (2 4 8 ...) should be used so a 3x3 resampling Gaussian kernel is selected.

JPEG_QUALITY_OVERVIEW 85 - if not specified the default of 75% is used, which does yield smaller file, but I find 85% a better compromise in the size vs quality trade off.

Update, 2015: GDAL 1.8 and 2.0 have introduced a lot of new options not covered here and which I haven't had time to digest. Read the official gtiff format page, I'm sure there are additional useful settings detailed.

10

For big rasters GeoTiff offers the possibility to store (pre-)downscaled overviews as extra images to the GeoTiff file. This can be done with gdaladdo (= GDAL ADD Overview). When creating these overviews, you can manually tell gdal to compress them too:

gdaladdo --config COMPRESS_OVERVIEW JPEG 

Speeds up viewing your data without adding too much size. Note: Geotools applications like Geoserver, uDig, AtlasStyler, Geopublisher can all use this feature and profit from overviews.

6

To enable partial image decompression, simply use TILED=YES .

Peter

4

Ultimately you'll probably need to experiment with the different options and see what meets your needs.

I've been making increased use of JPEG-compressed GeoTIFFs over wavelet-based formats. My results have been pretty good. Using GDAL to do this has yielded compression ratios comparable to wavelet-based formats without too much data loss. The performance hit that comes with decompression has been acceptable.

What I like most about this approach is that GeoTIFF support is almost universal, while support for wavelet-based formats isn't always assured and is sometimes subject to thorny licensing issues.

3

My experience comparing GeoTIFF vs. Earth Resource Mapping's ECW (Enhanced Compressed Wavelet) compression is that ECW is orders of magnitude better when compressing high resolution aerial photos. Another important advantage of wavelet based compression is that, unlike older formats as GeoTIFF, JPEG - not JPEG 2000 -, just a portion of the image can be decompressed [ref. 1]. The importance of this advantage must not be underestimated, specially when working with "larger than about half computer memory size.".

It seems - I have never had the chance of testing it - that MrSID, another propietary, wavelet based file format, also exhibits higher compression ratios than "older" formats and selective decompression.

ref. 1: http://www.ifp.uni-stuttgart.de/publications/phowo01/Ueffing.pdf

  • 1
    dariapra, remember that GeoTIFF-Packbits or GeoTIFF-LZW are lossless compressions while ECW and JPEG are lossy. Lossless or lossy compression must be carefully chosen depending on future data usage. – markusN Nov 16 '10 at 7:10
  • 1
    I am not claiming that a loosy compression format is always a valid storage format. What I wanted to mean is that using a format like ECW is suitable in some production environments. For instance, ECW is a more suitable format than GeoTIFF if we have a MapServer instance serving ortophoto layers via WMS. Nothing forbids that you also store the ortophoto using lossless compression. – dariapra Nov 16 '10 at 12:04
3

The answers by @dodobas and @matt-wilkie cover most everything relating to the acts of compressing and blurring with GDAL to reduce image size.

I would like to add two things:

  • the file-format documentation from GDAL: http://www.gdal.org/frmt_gtiff.html;
    • See the creation options (-co), specifically:
      • COMPRESS
      • NUM_THREADS
      • PREDICTOR
      • ZLEVEL
  • and that it is essential to verify that the software that will be consuming the GeoTIFFs:
    • supports the desired compression method;
    • recommends using compression.

For example, GeoServer does not recommend compressing GeoTIFFs:

As a final note, Geotiff supports various kinds of compression, but we do suggest to not use it. Whilst it allows for much smaller files, the decompression process is expensive and will be performed on each data access, significantly slowing down rendering. In our experience, the decompression time is higher than the pure disk data reading.

This is especially true if already using overviews, tiling, and high performance storage media (enterprise grade disk or SSD).

  • I also need to copress my jpeg image because I am not able to convert my raster to array with gdalin Python. It is showing memory error and also sometimes out of memory. Could anyone have any idea how can I implement this line (gdal_translate -co "COMPRESS=method" src_dataset dst_dataset) in python. I am new in using gdal. So, it is hard for me sometimes to understand the structure. – Shiuli Pervin Sep 22 '16 at 7:39
  • 1
    @ShiuliPervin, First, JPEG is already a compressed (lossy) format. Second, it sounds like you have an issue of chunk, not compression. Read the file in tiles, strips, or chunk, instead of all at once. Even if the file is compressed, it will have to be uncompressed when you use it (example: if a 4GB files uses 2GB on the disk when compressed, it will still take up 4GB of RAM when it is all loaded for processing. As a space saving alternative, you may want to look into sparse formatting for GeoTIFFs. – Kevin Sep 25 '16 at 15:53
  • 1
    @ShiuliPervin, Though, I may be misunderstanding your question. Compression itself often uses a lot of memory, but should not overflow your system, unless there is a bug in the library or you are given an invalid argument. If you are having issues with JPEG as the compression type for a GeoTIFF, maybe try LZMA or DEFLATE. – Kevin Sep 25 '16 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.