2

I am trying to run analysis on the yearly weather data files here: https://www.northwestknowledge.net/metdata/data/. I'm running into a problem where each yearly file for each variable requires 2.4GB memory if I read it in using ncdf4 package. I can use the raster packages brick function to create a pointer to the file on disk, but this is very slow.

How do I store a raster brick in my computer's memory, rather than just create a pointer? Also, does anyone have ideas on how to reduce the size of these files in R (they're about 100MB on disk)?

Improve this question

2 Answers 2

Reset to default
2

To load all values in RasterBrick into RAM you can use readAll

library(raster)
b <-brick("tmmn_2020.nc")
x <- readAll(b)

But this will not help you if you cannot have that much data in RAM, as you say.

Improve this answer
1

It is a little bit unclear what you want to do. Anyway you could use raster::stack() rather than raster::brick() to store multi-layer raster and saving time. While if you want to store the pointers with little memory usage you could read files as a raster stack or brick and save them as .rds file with saveRDS().

library(raster)
s<-stack("tmmn_2020.nc")
b<-brick(s)#this is slow
saveRDS(s,"stack.rds")
saveRDS(b,"brick.rds")
#read back
r <- readRDS("stack.rds")

To reduce file size on disk you can try to compress them while saving. Something similar to

writeRaster(s,"stack.tif",progress='text',option=c('COMPRESS=LWZ'))
writeRaster(b,"brick.tif",progress='text',option=c('COMPRESS=LWZ'))

But when you use compression, the biggest trade-off is extra processing time which is required to uncompress the image, and after uncompressing, the image would still consume the same amount of memory. In your case, however, it seems to me that the original files are already heavily compressed, I don't think you can do better with the raster package. There is the new terra package that implements lots of functionalities of the raster package, speeding up computation and reducing memory usage, but for your data, I think the best option is to lose some time to convert files to brick and saving as rds file with saveRDS(). For instance, I've downloaded the "tmmn_2020.nc" file (146 MB), which is a 366 multi-bands raster. Reading it in r with raster::stack() or terra::rast() it's instant, and saving the resulting file with saveRDS() took 0.1 seconds on my PC, resulting in an 8 Kb file (because it is a pointer).

library(terra)
t <- rast("tmmn_2020.nc")
saveRDS(t,"tstack.rds")
t <- readRDS("tstack.rds")
Improve this answer
2
  • If you can use raster::brick("file") then you should use it in favor of raster::stack("file") as it is more efficient. Also, it is not wise to use saveRDS with either type because, as you point out, you save a pointer to a file that may be no longer be around.
    – Robert Hijmans
    Commented Apr 19, 2021 at 0:35
  • And I should add that saving SpatRaster to a rds is always very bad advice. SpatRaster actually uses true "pointers" (i.e. a C++ construct) that are not valid after reading from a rds. So you really should never do that. With smaller files you could do saveRDS(pack(x)). (raster objects just store the filename, and as long the file still exists the object remains valid)
    – Robert Hijmans
    Commented Apr 19, 2021 at 5:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.