I would like to know what the smallest file format is I can use with GDAL that fulfills the following criteria:
- lossless compression
- (relatively) easy to read/write with GDAL Python bindings
- least HDD space used
- acceptable read/write performance
What have I tried so far? The best result so far seems to be a tiled GeoTiff with LZW compression:
gdal_translate -co "TILED=YES" -co "COMPRESS=LZW"
Why am I looking for this:
I am about to process time-series datasets which will take up around 6-8 TB in band sequential binary format. The processing is quite extensive but the output will only weigh in at around 30-40 GB. I know I am looking at a hit in read/write performance when using compression. An input/output speed degradation of around 300%-500% is acceptable - occupying 6-8 TB HDD space is not.
After the answer I wrote a small python script which tests the performance of different compressions and an input image of your choice and a little description - GeoTiff compression comparison