I work with large many (tens of thousands) of large (0.1-10 gigabytes) rasters in GDAL and Python and occasionally will see the following error:
ERROR 1: ZIPDecode:Decoding error at scanline [XXX], incorrect data check
This happens once every 10,000 files or so and I've been unable to reproduce it with a simple example. On the other hand it is reproducible on long computational runs.
I almost always work with rasters in compressed mode since their sizes are so large I'd otherwise be bottlenecked by the throughput to disk. I use the following GDAL creation options:
('TILED=YES', 'BIGTIFF=YES', 'COMPRESS=DEFLATE', 'BLOCKXSIZE=256', 'BLOCKYSIZE=256')
I have tried using
LZW compression as well as various predictors and found that that helps in some rasters but other rasters will fail. So far the only way I've been able to avoid this bug is to work with rasters in uncompressed mode, but it is undesirable since my computation is then often throttled by disk throughput.
If relevant, here's an example of a full error stack that happened to me a few minutes ago:
ERROR 1: ZIPDecode:Decoding error at scanline 11264, incorrect data check ERROR 1: TIFFReadEncodedTile() failed. ERROR 1: workspace_global_sdr_dir/sdr_workspaces/3/1/9/8/ws_na_bas_15s_beta_448913/intermediate_outputs/pit_filled_dem_ws_na_bas_15s_beta_448913.tif, band 1: IReadBlock failed at X offset 4\ 4, Y offset 6: TIFFReadEncodedTile() failed.
Does anyone have any experience with this type of error in GDAL, or any advice on how I should deal with large rasters, or any advice on how I could debug the issue?