2

I have several thousand small glacier rasters (in rectangular shape) that need to be clipped to the glacier's extent. I have the glacier polygons stored in a large SpatialPolygonsDataFrame (matching in length and position).

I need parallel computing or else it would take forever, so I would like to use the rslurm package.

My code so far (using the for-loop provided here: https://gis.stackexchange.com/a/342955/145570)

library(rslurm)

fun_clip_rslurm <- function(filelocations, filenames, glaciers){
  for(i in 1:length(filelocations)){
    r = raster(filelocations[i]) # get the ith file name
    g = glaciers[i,] # get the i'th row of `glaciers`.
    r = crop(r, g) # crop
    writeRaster(r, file.path("/data/.../RGI60-13_crop/", filenames[i]))
  }
}

params <- list(filelocations, filenames, glaciers)
names(params) <- c("filelocations", "filenames", "glaciers")

slurm_call(fun_clip_rslurm, params, jobname = "my_crop",
           slurm_options = list(time="1-00:00:00", share =TRUE), 
           submit = TRUE)

If I run this, I get the following error message:

sbatch: error: Batch job submission failed: Invalid account or account/partition combination specified
$jobname
[1] "my_crop"

$nodes
[1] 1

attr(,"class")
[1] "slurm_job"

So now I have two questions:
Do I have to give rslurm some kind of permission to run the batch-script on the cluster?
And how can I specify the number of nodes on which to run the calculation?

1 Answer 1

1

Okay, I found out what the problem was.

First: slurm_callwas not the right function, slurm_applywas more to the point, because I needed more than just one function call. With slurm_applyI can easily specify the number of nodes and cpus per node.
Secondly, I was having some permission problems that had nothing do to with my code and IT fixed it.
Now my code looks the following and runs like a charm:

fun_clip_rslurm <- function(filelocation, filename, glacier_index){
    r = raster(filelocation) 
    r = projectRaster(r, crs=my_CRS) # global object
    g = glaciers[glacier_index,] 
    r = crop(r, g)
    writeRaster(r, file.path("/data/.../", filename), overwrite = TRUE)
}

params <- data.frame(filelocation = filelocations, filename = filenames, glacier_index = 1:length(filenames),
                     stringsAsFactors = FALSE)

results <-  slurm_apply(fun_clip_rslurm, params, jobname = "DEM_crop",
            add_objects = c("glaciers", "my_CRS"), # feed in the global objects
            cpus_per_node = 20, nodes = 2000,
            slurm_options = list(time="0-01:00:00"), 
            submit = TRUE)

cleanup_files(results, wait = TRUE)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.