I have dealt with E-OBS gridded dataset and how to cropped raster grid for a particular country (I used 0.25-degree regular grid daily mean temperature observation). However, I have read about E-OBS grid dataset' publication and they told about maximum 20 percent of missing values might be occurred. After I cropped the raster grid for my interested regions and want to render plain text data with its metadata (geo-coordinate pair and daily level temperature observation), original missing values become 0.

It is important to me because it is difficult for me to differentiate which one is temperature observation (it could be 0 Celsius degree) and which one is missing observation. I want to render missing value with -999 or ....

To avoid this problem, I want to preserve missing values in a cropped raster grid, and treat those missing values when I intend to calculate the yearly average temperature for each coordinate of grid.

How can I deal with missing values in raster grid? Any idea?


I tried @Rodrigo' solution down below:

> tg1980 <- raster::brick("data/tg_0.25deg_reg_1980-1994_v17.0.nc")
> tg1980 <- reclassify(tg1980, cbind(NA, -999))
Error: cannot allocate vector of size 1.9 Gb

But now I got a memory problem in R. Any quick solution for that?

  • You could try to translate is.na(yourRaster) to some value out of the bounds, like -999. Then, after all conversions, crops, etc. return them to NA. – Rodrigo Apr 26 '18 at 15:30
  • I think it would be myraster[is.na(myraster)] <- -999. You don't need to specify rows and columns when changing all the values in a matrix or data.frame. – Rodrigo Apr 26 '18 at 15:38
  • @Rodrigo I just have this error: Error: cannot allocate vector of size 1.9 Gb. Why does this happen? How to fix that? Thank you – Andy.Jian Apr 26 '18 at 15:42
  • Looks like you're out of memory. Try rebooting, then open only R and try again. If this doesn't solve it, you could try the reclassify method below, or dividing your raster in smaller pieces, or using a more powerful machine. – Rodrigo Apr 26 '18 at 15:53

You should be able to use calc, in the raster package, to replace NA values via a function. However, please keep in mind that these are true nodata values along with missing values. This means that when you export the data, if you change these to real values, you will also be exporting the collar of the image representing the full extent. There are also other very good reasons, including statistical summaries, to have NA's as a place holder and not actual values. The index approaches are sound but do read the raster into memory so, are not always safe. Functions like calc will swap pixel blocks if the raster is too large to fit into memory.

r <- calc(r, function(x) { ifelse(is.na(x), -999, x) }, ... ) 

Note that I added the dot arguments so that you will look at the function help ?calc and use them. In this case they would be substituted for an output raster. Writing a raster to disk will mitigate the memory issue. Also, please make sure that you are not using the 32-bit version of R. By default both 32 and 64 bit versions are installed and you should be launching the "R x64" version.

As far as addressing missing values, this requires some sort of imputation. My preference is using a Local Polynomial Regression (loess) on the time series and filling in missing values based on the fit line. This is a well know method for smoothing timeseries data as well thus, removing some of the stochasticity before deriving moments. I have a function smooth.time.series in the development version of the spatialEco package that applies this method to a timeseries raster stack. The smooth.data argument controls whether the entire timeseries is smoothed (smooth.data=TRUE) or, if only missing values (NA's) are replaced (smooth.data=FALSE).

This example usage would replace NA values in the timeseries with values from the fit loess regression.

r2 <- smooth.time.series(r, f = 0.2, smooth.data = FALSE)    

I will reiterate, performing analysis, such as this, in software such as Excel is a very bad idea. Assuming that this data represents dailies for the time period 1980-1994 (which actually does not track with the number of layers you have) you could use the rts package to derive yearly summary statistics, or even just use an index call to the stack. Here is an example of what I am getting at.

( mean.1980 <- calc(r[[1:365]], na.rm = TRUE, fun=mean) )
  • You can speed things up a bit by playing with multithreding using beginCluster or changing the default rasterOptions settings for chunksize and maxmemory but, this will only buy you so much. The bottleneck is the number of rasters that you are processing. If you cannot process in-memory this is just a reality. I would recommend not removing NA's and figuring out your analysis using raster class objects. – Jeffrey Evans Apr 27 '18 at 19:38
  • The reason I want to treat missing value because I want to do yearly statistics for extracted Germany grid; if missing values are not correctly treated, it will cause a bias for calculating the yearly average temperature for each grid. Could you give me a possible workaround to overcome this problem? Many thanks – Andy.Jian Apr 27 '18 at 19:58
  • First of all, R deals with missing values which is why they exist as a special value (NA). In many R functions there is an argument on how to deal with NA values eg., mean(x, na.rm=T) and will in fact, return an NA if NA's occur in the data but are not addressed in the function argument. In addition, raster functions also provide NA handling arguments eg., raster::calc(x, na.rm = T, mean) It is time for you to find some online R training and work through it. It is apparent that you are trying to use a software that you do not know and as a result making erroneous assumptions. – Jeffrey Evans Apr 27 '18 at 20:13

Not the answer you're looking for? Browse other questions tagged or ask your own question.