Resource allocation to lidR LAScatalog process with SLURM bash script

Question

I have a possibility to do processing on a super computer, where task managing and resource allocation are controlled by SLURM (Simple Linux Utility for Resource Management) batch job system. However, I have not found the right configurations how to utilize the allocated resources with lidR efficiently. I have tried to allocate 20 CPU's to one task in SLURM and specified 20 workers for a multisession with Future-package within R script. After running a process for a short while, using LAScatalog processing engine, CPU efficiency statistics suggested that with these settings only one of the CPU's was used. Slurm bash script presented below

#!/bin/bash
#SBATCH --job-name=pointsToRaster
#SBATCH --account=project_num
#SBATCH --time=00:05:00
#SBATCH --output=output_%j.txt
#SBATCH --error=error_%j.txt
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=20
#SBATCH --mem-per-cpu=15G
#SBATCH --partition=hugemem

#A 5 MINUTE LONG TEST RUN

#load module
module load r-env-singularity

# Bind threads to individual cores
export OMP_PROC_BIND=true

#Run script
srun --threads-per-core=1 singularity_wrapper exec Rscript --no-save pointClouds.R

This bash script allocates resources and executes script pointClouds.R. Script reads in 30 .las files, containing point clouds produced with SFM-MVS photogrammetric methods. File sizes vary between 1Gt to 5Gt, and they are missing ground classification. First step is to classify groud points. Script content presented below.

#load packages

library(sf)
library(sp)
library(raster)
library(rgdal)
library(lidR)
library(future)

####### SET COMPUTATIONAL CONFIGURATIONS ##########

#Set working directory
setwd(dir = "/scratch/project_num/lasFiles")
filePaths = list.files(pattern = "./*las")

# Parallelization settings:
plan(multisession, workers = 20L)

#Read unclassified point clouds to a LAS-catalog object
pointCat = readLAScatalog(filePaths)

#### CLASSIFY GROUND POINTS ############

#Progressive Morphological Filter-algorithm 
opt_output_files(pointCat) = "./outputs/classified_{ORIGINALFILENAME}" 

ws  = seq(3, 12, 3) 
th  = seq(0.1, 1.5, length.out=length(ws))
groundClassified = lasground(pointCat, algorithm = pmf(ws, th))
rm(pointCat)

Tried changing the setting around determining 20 tasks per node and one CPU per task. This setting raised CPU utilization, but when looking at the "process outputs" -textfile, it shows that every part of the code was executed 20 times (i.e. every package was loaded 20 times). I am not sure are is the problem related to the bash or R-script.

Any help on how to correctly allocate resources with a SLURM bash script to achieve efficient parallel processing with lidR LAScatalog?

Answer 1

Issue was found by HPC service provider. For unknown reason OMP_PLACES=cores variable which should tie threads/processes to specific cores, appeared to bind all processes to a single core only when running multi-core R jobs. Issue has been solved by rebuilding r-environment singularity-container.

Answer 2

Your R script is correct. I think you should better use plan(multicore) since your are running the code on linux but this is not the issue. I think your issue is not related to lidR but to future and SLURM

I don't have any expertise with SLURM and I don't believe you will find anybody to help you on this GIS forum. You should better ask a question decoupled of lidR on stackoverflow where you will find more people with more advanced skills about this kind of stuff. The code that drives the processing of multiple files roughly looks like that. This will help you to make some tests yourself and ask a more focused question to more qualified people.

library(future)

custom_apply = function(n = 10)
{
  f = vector("list", n)
  for (i in 1:n)
  {
    f[[i]] = future({
      # do something computationally demanding
      # here I only sleep for a random delay
      d = runif(1, 0, 1)
      Sys.sleep(d)
      return (d)
    })
  }
  return(values(f))
}
  
plan(sequential)

ti = Sys.time()
custom_apply(10)
Sys.time() - ti
#> Time difference of 6.39 secs

plan(multisession, workers = 4)

ti = Sys.time()
custom_apply(10)
Sys.time() - ti
#> Time difference of 2.51 secs