0

I have a code where I read in bricks for precipitation, min and max temperature, calculate bioclimatic variables from them, and then write rasters with bioclimatic variables for the time slice of the brick:

library(raster)
library(dismo) 
library(parallel)

setwd("/scratch/bntjoa002/chelsa_cmip5_ts/")
options(rasterTmpDir = "/scratch/bntjoa002/rtmp")

# set the number of cores
mc.cores=20

#get model names
modelnames<-c("MIROC5")

#experiment names
experiments<-c("rcp45", "rcp85")

#specify years
start.year<-2030
end.year<-2049

# start the loop to calculate bioclim vars

for (i in 1:length(modelnames)){
  
  model = modelnames[i]
  
  for(j in 1:length(experiments)){
    
    experiment<-experiments[j]
    
    tasmax<-brick(paste0(getwd(),'/tasmax','/CHELSAcmip5ts_tasmax_',model,"_",experiment,"_",start.year,"-",end.year, ".nc"),
                  varname='air_temperature')
    tasmax<-tasmax-273.15
    
    tasmin<-brick(paste0(getwd(),'/tasmin','/CHELSAcmip5ts_tasmin_',model,"_",experiment,"_",start.year,"-",end.year, ".nc"),
                  varname='air_temperature')
    tasmin<-tasmin-273.15
    
    pr<-brick(paste0(getwd(),'/pr','/CHELSAcmip5ts_pr_',model,"_",experiment,"_",start.year,"-",end.year, "_V1.1.nc"),
              varname='precipitation_flux')
    pr<-pr*24*60*60*30
    
    # make a vector with the years in climate data time series
    years<-seq(start.year,end.year,1)
    
    nyears=length(years)
    
    # make index for the year
    ind=sort(rep(1:nyears,12))
    
    # Calculate bioclim variables for each year using mclapply
    bcbricks=mclapply(unique(ind),function(x){
      #extract just the climate data for each year from the raster brick
      tasmax.year=tasmax[[which(ind==x)]]
      tasmin.year=tasmin[[which(ind==x)]]
      precip.year=pr[[which(ind==x)]]
      # calculate bioclim variables
      biovars(precip.year,tasmin.year,tasmax.year)
    },mc.cores=mc.cores)
    
    
    #if you want to save one bioclim var as a stack across all years
    if(experiment=="rcp45"| experiment=="rcp85"){
      
      # output directory
      outDir=paste0('/scratch/bntjoa002/chelsa_cmip5_ts/MIROC5')
      
      # stack individual bioclim variables across years
      lapply(1:nlayers(bcbricks[[1]]),function(x){
        
        bcvar.by.allyears=stack(lapply(bcbricks,function(y){ y[[x]]}))
        
        writeRaster(bcvar.by.allyears,file=paste0(outDir,'/',names(bcbricks[[1]])[x],'_',model,'_',experiment,'_',start.year,'_',end.year,'.tif'))
      })
    }
    print(j)
  }
  print(i)
} 

In this instance (for the years 2030--2049), the code works for the first 22 files (producing bioclimatic variables 1 to 19 for rcp45 and 1 to 3 for rcp85), following which I get the error:

Error in rgdal::putRasterData(x@file@transient, vv, band = i, offset = off) : 
  Failure during raster IO
Calls: lapply ... writeValues -> writeValues -> .local -> <Anonymous>
Execution halted

I know that the code in general works because it has worked for other time slices using data from the same source (though I do get this error for a few of the other time slices too). The code seems to have no problem making the bricks and calculating the bioclim variables, but does have an issue when writing the results. If I open the bricks in r for each of pr, tasmax and tasmin, the bricks look fine (same dimensions etc.).

Any hints on how I can go about troubleshooting this?

3
  • It probably means that you have run out of disk space Commented Nov 27, 2020 at 18:16
  • Thanks @RobertHijmans I thought it might be a problem for the temp files but I checked the drive where the temp files are being saved and there is still at least 5 TB free space. And, the drive where the rasters are being written is the same drive as where the temp files are bring saved. Commented Nov 27, 2020 at 18:48
  • 1
    It might be helpful to not use lapply at the end but just loops, that makes it easier to see what is going on, and to print messages (e.g. filenames) that can help you debug. If would make the filename a variable, and print it. Commented Nov 27, 2020 at 19:27

1 Answer 1

0

Suspecting that this might be a RAM issue, following @Robert Hijmans advice, we changed lapply at the end to a for loop that reads in the rasters individually as follows :

  if(experiment=="rcp45"| experiment=="rcp85"){
      
      outDir=paste0('/scratch/bntjoa002/chelsa_cmip5_ts/', model)
      
      years<-seq(start.year,end.year,1)
      for(l in 1:19){
        bioname<-paste0("bio",l)
        for (k in 1:length(bcbricks)){
          year=years[k]
          bioclimras<-bcbricks[[k]][[l]] 
          writeRaster(bioclimras, file=paste0(outDir,'/',bioname,"/",names(bcbricks[[1]])[l],'_',model,'_',experiment,'_',year,'.tif'))
          print(paste0("k_",k))
          
        }
        print(paste0("l_",l))
      }
    }

And we no longer receive the rgdal::putRasterData error, suggesting that this was indeed a RAM issue.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.