I work with large many (tens of thousands) of large (0.1-10 gigabytes) rasters in GDAL and Python and occasionally will see the following error:

ERROR 1: ZIPDecode:Decoding error at scanline [XXX], incorrect data check

This happens once every 10,000 files or so and I've been unable to reproduce it with a simple example. On the other hand it is reproducible on long computational runs.

I almost always work with rasters in compressed mode since their sizes are so large I'd otherwise be bottlenecked by the throughput to disk. I use the following GDAL creation options: ('TILED=YES', 'BIGTIFF=YES', 'COMPRESS=DEFLATE', 'BLOCKXSIZE=256', 'BLOCKYSIZE=256')

I have tried using LZW compression as well as various predictors and found that that helps in some rasters but other rasters will fail. So far the only way I've been able to avoid this bug is to work with rasters in uncompressed mode, but it is undesirable since my computation is then often throttled by disk throughput.

If relevant, here's an example of a full error stack that happened to me a few minutes ago:

ERROR 1: ZIPDecode:Decoding error at scanline 11264, incorrect data check
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: workspace_global_sdr_dir/sdr_workspaces/3/1/9/8/ws_na_bas_15s_beta_448913/intermediate_outputs/pit_filled_dem_ws_na_bas_15s_beta_448913.tif, band 1: IReadBlock failed at X offset 4\
4, Y offset 6: TIFFReadEncodedTile() failed.

Does anyone have any experience with this type of error in GDAL, or any advice on how I should deal with large rasters, or any advice on how I could debug the issue?

  • Can you edit your question to include a small code snippet that shows the sort of operations you are doing. I realise it may not reproduce the error. – user2856 Jun 11 '19 at 4:57