2

I have a raster I'm trying to extract, the projections of my shapefile and raster are the same:

ext <- raster::extract(ras.tiftest2, single, na.rm=TRUE, fun=sum)

I get this error:

Error in rgdal::getRasterData(con, offset = offs, region.dim = reg, band = 
object@data@band) : 
Failure during raster IO

I found this post (Why do I get "Error: Failure during raster IO" when extracting values from my raster stack?) which suggested throwing it into QGIS and saving out of QGIS which worked like a charm, except I have about 200 rasters I'd like to do so it isn't feasible to save each one in QGIS.

After saving through QGIS:

ext <- raster::extract(qgisBGD, single, na.rm=TRUE, fun=sum)

> ext
     [,1]
[1,] 149786.1

IS there a way to mimic in R whatever QGIS is doing with the file to fix it?


When I save with QGIS I A. get a tiff file that is over twice as big as the original (76MB vs. the saved QGIS file is 160MB), and B. get an .aux.xml file of the same name as the tif. The content of the xml file contains HISTMIN, HISTMAX, HISTCOUNTS, BUCKET COUNT, STATS MAX, STATS MEAN, STATS MINIMUM, STATS STDDEV. However this doesn't seem to be essential as I tested the dropbox version below without the other .xml file in a new directory.

The raster files that work vs not are in the below dropbox link. Clearly different even based on the preview in dropbox. They are the same exact file, I just saved one through QGIS and didn't the IOERROR one. Not sure how/why QGIS is doubling the file size, I'm more of an R guy. I tested these files off downloads from the dropbox file, they function with the exact errors (or non-errors) above. Loaded the IO error one into QGIS, suddenly twice as big and works like a charm.

Files: https://www.dropbox.com/sh/jr1mybbwntq2zm7/AAAQPyfnK3ud6zVDx0sZ77BDa?dl=0


@Spacedman request

> print(REGRAS)
class       : RasterLayer 
dimensions  : 7271, 5607, 40768497  (nrow, ncol, ncell)
resolution  : 0.0008333, 0.0008333  (x, y)
extent      : 88.00824, 92.68055, 20.57565, 26.63457  (xmin, xmax, ymin, ymax)
coord. ref. : +proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0 
names       : WPREGRASTER2 
values      : 0, 934.4766  (min, max)

> print(QGISRASTER)
class       : RasterLayer 
dimensions  : 7271, 5607, 40768497  (nrow, ncol, ncell)
resolution  : 0.0008333, 0.0008333  (x, y)
extent      : 88.00824, 92.68055, 20.57565, 26.63457  (xmin, xmax, ymin, ymax)
coord. ref. : +proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0 
names       : QGISTTT 

> summary(REGRAS)
Error in rgdal::getRasterData(con, offset = offs, region.dim = reg, band = 
object@data@band) : 
Failure during raster IO

> summary(QGISRASTER)
        QGISTTT
Min.       0.000000
1st Qu.    2.797436
Median     5.271086
3rd Qu.    9.715539
Max.    1247.123779
NA's       0.000000
  • Any chance we can get our hands on these raster files? Any chance you can show us the output of print and summary on your raster objects in both cases? Are the rasters coming from geoTIFFs, ESRI Grid files, JPEGs with world files? Does the raster plot okay in R? I think we need more information... – Spacedman Jan 29 '18 at 8:04
  • See above, the REGRAS (the non-QGIS saved one) plots just fine, they are GTiffs – Neal Barsch Jan 29 '18 at 8:36
  • If you would like the raster geotiff files, one that was saved by QGIS vs. not (I literally loaded the same file, after changing the name of the IO error one into QGIS) here:dropbox.com/sh/jr1mybbwntq2zm7/AAAQPyfnK3ud6zVDx0sZ77BDa?dl=0 – Neal Barsch Jan 29 '18 at 9:34
  • What're your operating system, rgdal and QGIS versions? – Simbamangu Jan 29 '18 at 11:13
2

Don't trust the original raster file.

When I tried converting it on the command line, I got:

$ gdal_translate -co "COMPRESS=NONE" ./IOERROR.tif IOERRORgdal.tif
Input file size is 5607, 7271
0...10...20...30...40...50...60...70...80...90...ERROR 1: TIFFFillTile:Read error at row 4992, col 6656; got 1817 bytes, expected 2564
ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: ./IOERROR.tif, band 1: IReadBlock failed at X offset 40, Y offset 56
ERROR 1: GetBlockRef failed at X block offset 40, Y block offset 56

So it's not R's fault - R uses gdal, and gdal here didn't like it. So why did QGIS like it? I tried loading it into QGIS, and QGIS didn't like it that much either, throwing some warnings:

Warning: RasterIO error: /home/rowlings/Downloads/Errors/IOERROR.tif, band 1: IReadBlock failed at X offset 40, Y offset 56

Something in that file is wrong, and I wouldn't trust the TIFF saved from QGIS either.

Solution

With that massive caveat lodged firmly in your cranium, it is possible to get the GDAL utilities to produce a working version of your file, using gdalwarp on the command line:

$ gdalwarp  IOERROR.tif IOERROR-fix.tif
Creating output file that is 5607P x 7271L.
Processing input file IOERROR.tif.
Using internal nodata values (e.g. -999) for image IOERROR.tif.
Copying nodata values from source IOERROR.tif to destination IOERROR-fix.tif.
0...10...20...30...40...50...60...70...80..ERROR 1: TIFFFillTile:Read error at row 4992, col 6656; got 1817 bytes, expected 2564
ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: IOERROR.tif, band 1: IReadBlock failed at X offset 40, Y offset 56
ERROR 1: GetBlockRef failed at X block offset 40, Y block offset 56

Which I can read and summary into R:

> IF = raster("./IOERROR-fix.tif")
> summary(IF)
         IOERROR.fix
Min.    3.488778e-02
1st Qu. 3.041900e+00
Median  5.504765e+00
3rd Qu. 9.975844e+00
Max.    1.914248e+03
NA's    0.000000e+00

and it plots too.

The reason the QGIS file is so large is because it isn't using any compression - as does the above command, but you can change this with:

 gdalwarp  -co "COMPRESS=LZW" IOERROR.tif IOERROR-fix-lzw.tif

which returns a smaller file that still works in R.

GDAL utils package

If you struggle with command line stuff, there's an R package that wraps them and might help: https://cran.r-project.org/web/packages/gdalUtils/index.html

  • I realized I forgot to get back to you on this. Your solution works, but it pointed me in the right direction for a solve where the file didn't come in corrupted. Definitely a good warning lodged in my cranium! :) I was using writebin to write a binary file into memory. I realize now this is kinda dumb for the file size I was working with, and switched to write_disk to see if it resolved the issue entirely. It did. That being said it's nice to see what QGIS was doing, so thanks for the work and my bad on not realizing my memory was corrupting the large file. – Neal Barsch Feb 26 '18 at 4:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.