R rasterEngine Parallel Processing, serialized and unserialized memory errors

Question

I am trying to perform some raster math and wanted to try parallel processing in R using rasterEngine.

I am able to run the script for a few minutes and watch my CPU usage increase substantially, but soon an error occurs and processing stops.

Here is my code

library(raster)
library(spatial.tools)
library(doParallel)

cpus = 24 # The number of nodes/cores to use in the cluster
cl <- makeCluster(spec = cpus, type = 'PSOCK', methods = FALSE, outfile="E:/Heatload_10m/cluster.txt")
registerDoParallel(cl)

l = raster('E:/Heatload_10m/Lat_Radian.img')
s = raster('E:/Heatload_10m/R1_10m_WGS84_Slope.img')
f= raster('E:/Heatload_10m/Folded_Aspect.img')

heatload <- function(lat,slope,folded){
    hl= exp(-1.467 + 1.582*cos(lat)*cos(slope)-1*cos(folded)*sin(lat)*sin(folded)-0.262*sin(lat)*sin(slope)+0.607*sin(folded)*sin(slope))
    return (hl)
}

print('Starting Heatload Calc')
final_heatload <- rasterEngine(lat=l,slope=s,folded=f, fun=heatload, overwrite=TRUE, processing_unit = "chunk", chunk_format = "raster" setMinMax =TRUE, )
stopCluster(cl) 
registerDoSEQ()  

Here are the error messages:

The instruction at 0x61b83401 referenced memory at 0xd120c1d0. The memory could not be written.

Error in unserialize(socklist[[n]]) : error reading from connection

Error in serialize(data, node$con) : error writing to connection

Answer 1

This is an old question, so hopefully you found the answer. This doesn't address your question but after setMinMax = TRUE you need to delete the comma.

Regarding your question, I think that you should specify an output file by setting filename = yourpath.tif. Then rasterEngine will use that as a temporary file. The alternative is it will likely set a temp file somewhere in your C drive, which can not only lead to read/write issues but also to memory issues (e.g. if your C drive is quite small as is my case, then rasterEngine will go on forever and not crash). I think your writing issue might have to do with that.

In any case, you should be testing your function using debugmode = TRUE to see where exactly your code is crashing. If your function code runs ok, then set verbose = TRUE and have a look at when the function is crashing and what it tried to do beforehand.