efficient workaround to treat missing values in gridded data in `RasterBrick` object [duplicate]

Question

I have dealt with E-OBS gridded dataset and how to cropped raster grid for a particular country (I used 0.25-degree regular grid daily mean temperature observation). However, I have read about E-OBS grid dataset' publication and they told about maximum 20 percent of missing values might be occurred. After I cropped the raster grid for my interested regions and want to render plain text data with its metadata (geo-coordinate pair and daily level temperature observation), original missing values become 0.

It is important to me because it is difficult for me to differentiate which one is temperature observation (it could be 0 Celsius degree) and which one is missing observation. I want to render missing value with -999 or ....

To avoid this problem, I want to preserve missing values in a cropped raster grid, and treat those missing values when I intend to calculate the yearly average temperature for each coordinate of grid.

How can I deal with missing values in raster grid? Any idea?

Update:

I tried @Rodrigo' solution down below:

> tg1980 <- raster::brick("data/tg_0.25deg_reg_1980-1994_v17.0.nc")
> tg1980 <- reclassify(tg1980, cbind(NA, -999))
Error: cannot allocate vector of size 1.9 Gb

But now I got a memory problem in R. Any quick solution for that?

Answer 1

You should be able to use calc, in the raster package, to replace NA values via a function. However, please keep in mind that these are true nodata values along with missing values. This means that when you export the data, if you change these to real values, you will also be exporting the collar of the image representing the full extent. There are also other very good reasons, including statistical summaries, to have NA's as a place holder and not actual values. The index approaches are sound but do read the raster into memory so, are not always safe. Functions like calc will swap pixel blocks if the raster is too large to fit into memory.

r <- calc(r, function(x) { ifelse(is.na(x), -999, x) }, ... ) 

Note that I added the dot arguments so that you will look at the function help ?calc and use them. In this case they would be substituted for an output raster. Writing a raster to disk will mitigate the memory issue. Also, please make sure that you are not using the 32-bit version of R. By default both 32 and 64 bit versions are installed and you should be launching the "R x64" version.

As far as addressing missing values, this requires some sort of imputation. My preference is using a Local Polynomial Regression (loess) on the time series and filling in missing values based on the fit line. This is a well know method for smoothing timeseries data as well thus, removing some of the stochasticity before deriving moments. I have a function smooth.time.series in the development version of the spatialEco package that applies this method to a timeseries raster stack. The smooth.data argument controls whether the entire timeseries is smoothed (smooth.data=TRUE) or, if only missing values (NA's) are replaced (smooth.data=FALSE).

This example usage would replace NA values in the timeseries with values from the fit loess regression.

r2 <- smooth.time.series(r, f = 0.2, smooth.data = FALSE)    

I will reiterate, performing analysis, such as this, in software such as Excel is a very bad idea. Assuming that this data represents dailies for the time period 1980-1994 (which actually does not track with the number of layers you have) you could use the rts package to derive yearly summary statistics, or even just use an index call to the stack. Here is an example of what I am getting at.

( mean.1980 <- calc(r[[1:365]], na.rm = TRUE, fun=mean) )
  plot(mean.1980)