Dividing a large raster into blocks for iteration in R

Question

I have very big rasters for processing (#pixels >1,5*10^10) for this project I am working on. Simple functions such as finding the area of a pixels with a particular value, across the entire raster 1) takes a lot of time and more importantly 2) is unsuccessful because of in_memory problems (yes, I have set rasterOptions(); chunksize() and memorysize())

I think the best ways forward is to divide the raster into blocks and then iterate through these blocks using the doParallel and forEach packages.

I took a stab and am still hitting the same error

Error in rgdal::getRasterData(con, offset = offs, region.dim = reg, band = object@data@band) : long vectors not supported yet: memory.c:3451

This is my code

bs<-blockSize(big_raster) 
bs$n #16 blocks
bs$nrow #gives row number at which a chunk can be made         

library(doParallel)
library(foreach)

UseCores<-detectCores() #detect number of cores
UseCores #24

cl<-makeCluster(UseCores)
registerDoParallel(cl)

length(cl)

x<-list() #to store areas for different chunks as a list
foreach(ras=1:big_raster[bs$nrow,]) %dopar% {
library(raster)
x[]<-((sum(ras[]==3)*(res(ras) [1])^2)/10000)/10^6 #Area in million ha of all pixels of value 3 in big_raster (linear unit is m)
}
stopCluster(cl)    

Answer 1

It is difficult to understand the memory problems you report as you do not show the code that causes it. Perhaps you do something wrong. It could also be useful to see the results of

show(big.raster)

and

canProcessInMemory(big.raster, 4, TRUE)

(this would look something like this)

#memory stats in GB
#mem available: 53.67
#        60%  : 32.2
#mem needed   : 0
#max allowed  : 4.66  (if available)

As for parallelization, the raster package has some support built in via the clusterR method. Your approach is doomed to fail for a large raster as you store all values in memory.