4

I would like to know what the smallest file format is I can use with GDAL that fulfills the following criteria:

  • lossless compression
  • (relatively) easy to read/write with GDAL Python bindings
  • least HDD space used
  • acceptable read/write performance

What have I tried so far? The best result so far seems to be a tiled GeoTiff with LZW compression:

gdal_translate -co "TILED=YES" -co "COMPRESS=LZW"

Why am I looking for this:

I am about to process time-series datasets which will take up around 6-8 TB in band sequential binary format. The processing is quite extensive but the output will only weigh in at around 30-40 GB. I know I am looking at a hit in read/write performance when using compression. An input/output speed degradation of around 300%-500% is acceptable - occupying 6-8 TB HDD space is not.

edit:

After the answer I wrote a small python script which tests the performance of different compressions and an input image of your choice and a little description - GeoTiff compression comparison

4

You should read the whole thread of http://thread.gmane.org/gmane.comp.gis.gdal.devel/38725.

Test image was an aerial photo (424 MB) and methods with best compression were:

  1. Lossless JPEG2000 (197 MB)
  2. DEFLATE compression with PREDICTOR=2 (280 MB)
  3. LZW compression with PREDICTOR=2 (307 MB)

However, your images seem to suit very well for LZW compression and probably for DEFLATE as well which means that JPEG2000 may not be the best option for you. Also, the PREDICTOR=2 setting gives better compression for natural images with lots of tones but in your images the neighrouring pixels may not have correlation and then PREDICTOR=1 should lead to better compression. You have perhaps already found a a good method but make a few more tests with deflate and different predictor values.

GDAL, since two days, supports also reading of Better Portable Graphics (bpg) format http://bellard.org/bpg/. There are very little experience on this format but it does have interesting features.

  • 1
    Excellent source! I ran some tests over 10 MOD09Q1 and MOD13Q1 images. The smallest filesize was achieved by COMPRESS=DEFLATE PREDICTOR=1. It took around 3x longer to write than uncompressed and 1x longer than LZW compression. Since mine is the case of write once, read often I checked the read times. uncompressed: 465 ms per loop, LZW, predictor 2: 2.2 s per loop, DEFLATE, predictor 2: 1.9 s per loop, DEFLATE, predictor 1: 1.62 s per loop. In my case -co "COMPRESS=DEFLATE" -co "PREDICTOR=1 is a clear winner with a size reduction of 65% and read time increase of less than 400%. – Kersten Feb 2 '15 at 9:18

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.