0

I have a raster, which is not very big, only a few hundred megabytes compressed, but it has about 10 billion values, each stored in 1 byte (0..255). Uncompressed it would take about 10gb (lots of empty cells). I had a different issue when I realized that something like

library(raster)
myRaster <- raster("myRasterFile.tiff") # <--this file is not more than a few hundred mb
myRaster[1] <- 5

takes a very long time, without even using the disk, or the RAM too much. Just one core of the CPU is maxed and that's all. I guess the raster is not loaded into memory, since I don't get a 10Gb spike in RAM usage when I do myRaster <- raster("myRasterFile.tiff"), and I would also guess that R knows where my value is located, since it shows me the value of myRaster[1] instantly.

I thought that maybe the raster package might not want to mess with my file before I use writeRaster, so then for any write operation it would load the whole file into memory, but since there's no spike in the memory (it actually uses very little), it means it probably doesn't do that. After that I thought that maybe it first uncompresses the whole file to the disk, bit by bit to a temporary location, and then it either loads it into ram, or use it from the disk. The CPU usage would make sense for uncompressing, although how does it get the first value to begin with without uncompressing. Anyway, this can't be true either since rsession doesn't even touch the disk and it doesn't increase the memory usage.

Any ideas of why this is happening and maybe solutions so that it doesn't happen?

Improve this question
4
  • More than one hour passed since I run the code above and it's still running, albeit nowhere and I swear no nazi codes were decrypted on this very computer. I can't possibly fathom what it is going on in its little consumer level CPU of it.
    – Andrei
    Commented Mar 14, 2018 at 16:29
  • 1
    Make a sequence of tiffs of different sizes up to this problematic one and see how long it takes to do the same thing to each of them.
    – Spacedman
    Commented Mar 15, 2018 at 10:11
  • What are you trying to accomplish, exactly? Just set a single upper-left corner value of myRaster to 5 (which is what your code does)?
    – Simbamangu
    Commented Mar 19, 2018 at 7:03
  • @Simbamangu I am trying to assign some raster cells to an expression a bit more complex. Whether I use the expression or just 5 makes no difference for the computation time. Also, I will assign values one by one, because myRaster[(expression)] tries to put 40gb of data in ram, which is slightly over my capacity.
    – Andrei
    Commented Mar 19, 2018 at 12:07

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.