My algorithm takes a global dataset at 30m resolution, breaks it up into pieces, analyzes the pieces somehow, and writes the analyzed piece to disk. As all the pieces are independent, this is embarrassingly parallel.
I am using the joblib package to parallelize tasks. I have two global datasets: vrt.vrt (virtual raster) and tiff.tiff (geotiff). When I run a parallelized task that reads from vrt.vrt, I have no problems. However, when I run a parallelized task that reads from tiff.tiff, I get the following error message:
ERROR 1: LZWDecode:Wrong length of decoded string: data probably corrupted at scanline 368132 ERROR 1: TIFFReadEncodedStrip() failed. ERROR 1: /path/landmasses.tif, band 1: IReadBlock failed at X offset 0, Y offset 368132 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 368132
The read command is the same for vrt.vrt and tiff.tiff:
tiff = gdal.Open('tiffpath') start parallel tiledata = tiff.ReadAsArray( xoff = int(xmin-npad), yoff = int(ymin-npad), xsize = int(ncol+2*npad), ysize = int(nrow+2*npad) )
However, when I run the tiff.tiff in serial, there are no errors and the output files look correct.
So does anyone know why gdal can't read from a .tiff using multiple threads, but can for a virtual raster?
In effort to progress, I built a virtual raster from the single geotiff file, and parallelization works fine. The question remains, however.