I have a raster, which is not very big, only a few hundred megabytes compressed, but it has about 10 billion values, each stored in 1 byte (0..255). Uncompressed it would take about 10gb (lots of empty cells). I had a different issue when I realized that something like
library(raster) myRaster <- raster("myRasterFile.tiff") # <--this file is not more than a few hundred mb myRaster <- 5
takes a very long time, without even using the disk, or the RAM too much. Just one core of the CPU is maxed and that's all.
I guess the raster is not loaded into memory, since I don't get a 10Gb spike in RAM usage when I do
myRaster <- raster("myRasterFile.tiff"), and I would also guess that R knows where my value is located, since it shows me the value of
I thought that maybe the raster package might not want to mess with my file before I use
writeRaster, so then for any write operation it would load the whole file into memory, but since there's no spike in the memory (it actually uses very little), it means it probably doesn't do that.
After that I thought that maybe it first uncompresses the whole file to the disk, bit by bit to a temporary location, and then it either loads it into ram, or use it from the disk. The CPU usage would make sense for uncompressing, although how does it get the first value to begin with without uncompressing. Anyway, this can't be true either since rsession doesn't even touch the disk and it doesn't increase the memory usage.
Any ideas of why this is happening and maybe solutions so that it doesn't happen?