I have a raster, which is not very big, only a few hundred megabytes compressed, but it has about 10 billion values, each stored in 1 byte (0..255). Uncompressed it would take about 10gb (lots of empty cells). When I try getValues(myRaster) I get an error that it cannot allocate vector of size 40Gb. Judging by the number, I get it that it tries to put everything in a vector with 4 bytes values. This means that in just one line, when I try to process a 300mb file, I get an error that I need 40gb of ram. Here's some example code:

myRaster <- raster("myRasterFile.tiff") # <--this file is not more than a few hundred mb
#Error: cannot allocate vector of size 40 Gb

I do understand that R is not made with programmers in mind. But is there any way I can force getValues(myRaster) to return a vector coerced as 1 byte values?

I know I can just split the raster into whatever fits my RAM, and maybe even run things in parallel as many times as it fits my CPU, but that code would not showcase at all the simplicity and elegance of R for statistics.

I also know I can just increase the swap temporarily to more than 40Gb, thus the virtual RAM, and then just wait until it's done, but this code will literally work only on my computer.

Does anyone have any idea of how would this can be solved?

  • Just don't use getValues, that's for getting every value in one go. There are many other functions for extracting values by index, by point overlay, line, polygon overlay and so on. There's also a lot of documentation around efficient ways to use raster, but I can't see here what you want to do. – mdsumner Mar 14 '18 at 14:44
  • that's fair, but what I actually need is something like myRaster[myRaster[]>30] <- 5. The error shows up at myRaster[], which is short for values(myRaster). I didn't want to clog the question with irrelevant information, since I knew I need to go through all the values nevertheless – Andrei Mar 14 '18 at 14:50
  • try this for general advice vignette("functions", package= "raster") - but do see that getValues() has arguments row and nrows which gives chunk-wise access pretty clearly. It will be tough to reduce the memory footprint given you have 1 byte, R can't really do that unless you use raw() - I'm not sure if raster can pass that through, so it will dictate your chunking. You might even be better to read with more generic TIFF readers, but raster's pretty powerful once you know your way around – mdsumner Mar 14 '18 at 14:54
  • Thanks! I am quite familiar with most raster functions. At the end of the day I need to get through every value, compare it, and then update it. There's probably no way around splitting everything in chunks that fit into memory. Now I have a new problem. On the same raster I discovered that even doing something as basic as myRaster[1] <- 5 takes a lot of time. I'll ask another question about this. – Andrei Mar 14 '18 at 15:17
  • ok cool, this is a good question - I was just a bit surprised you expected getValues() to be fine! That has to decompress the raw data and expand it, and I can't think of a way around doing that in chunks. Another important thing, is the tiff tiled? Please try rgdal::GDALinfo("myRasterFile.tiff") and report, the rows/columns vs. blockSize1 blockSize2 will tell us (raster is slow with some tile things). Also just print(myRaster) will be helpful. (rgdal might be better overall, but I haven't tried writing out to an existing file) – mdsumner Mar 14 '18 at 20:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.