Any workaround to extract plain text data in ASCII from `netCDF` file in R

Question

I am wondering about how to proportionally split or subset the gridded dataset in netCDF format where data sources were provided by European Climate Assessments and dataset (link for data source). However, E-OBS climate gridded dataset were given for each 15 years climate observation throughout Europe and working with big netCDF format grid data in R always cause memory problem in my site when I tried to treat all missing values. After decompressing grid data in netCDF format (download link of grid data), I used raster::stack to import whole gridded data, then I tried to treat missing values in this grid data because I need to do some statistics on this temperature observation later on, but it caused memory problem. So here is what in R:

mydat <- raster::stack("~/data/tg_0.25deg_reg_1980-1994_v17.0.nc")
> print(mydat)
File C:\Users\jvrat\Documents\stella\data\tg_0.25deg_reg_1980-1994_v17.0.nc (NC_FORMAT_CLASSIC):

     1 variables (excluding dimension variables):
        short tg[longitude,latitude,time]   
            long_name: mean temperature
            units: Celsius
            standard_name: air_temperature
            _FillValue: -9999
            scale_factor: 0.00999999977648258

     3 dimensions:
        longitude  Size:464
            long_name: Longitude values
            units: degrees_east
            standard_name: longitude
        latitude  Size:201
            long_name: Latitude values
            units: degrees_north
            standard_name: latitude
        time  Size:5479   *** is unlimited ***
            long_name: Time in days
            units: days since 1950-01-01 00:00
            standard_name: time

    5 global attributes:
        Ensembles_ECAD: 17.0
        Conventions: CF-1.4
        References: http://www.ecad.eu\nhttp://www.ecad.eu/download/ensembles/ensembles.php\nhttp://www.ecad.eu/download/ensembles/Haylock_et_al_2008.pdf
        history: Wed Apr  4 11:17:28 2018: ncks -a -d time,10957,16435 tg_0.25deg_regular_1.nc tg_0.25deg_reg_1980-1994_v17.0.nc
Wed Apr  4 11:16:02 2018: ncks -a --mk_rec_dmn time tg_0.25deg_regular.nc tg_0.25deg_regular_1.nc
        NCO: 4.6.7

mydat <- raster::reclassify(mydat, cbind(NA, -999))
Error: cannot allocate vector of size 2.9 Gb

To avoid this memory problem, perhaps I can try to split original gridded data in netCDF format in 10 years and 5 years observation or equally split the grid data by each 5 years climate data observation (originally it was 15 years climate data observation for whole Europe). But doing this way is not feasible and desired.

UPDATE: objective:

I want to either convert this netCDF file to plain text tabular data in ASCII format (extract plain text data of Germany grid from chosen E-OBS netCDF grid data). I can use R to work with this data manipulation, then I will use its output data (plain text data in ASCII) in ArcGIS desk for different interpolation.

UPDATE 2:

I need to extract plain text data of Germany grid with 0.25 degree resolution from original E-OBS netCDF files where 15 years climate observation for whole Europe were stored in each nc file. I need all grid data in ASCII format or csv format because I can import them with ArcGIS desktop and apply respective geospatial analysis. I need to find a solution how to get this task done that I specified above. Any idea please?

How can I make this happen this extraction or possible coercion of netCDF data? Is that doable in R? Any more thoughts?

Answer 1

Are you going to work with these data in R? Then only do the resetting of values with the a slice at a time. See how mydat[[1]] compares to mydat.

Are you going to work with it as a NetCDF elsewhere? R's raster is not a good tool for creating a new NetCDF file or modifying one. See ncdf4 or RNetCDF package for that.

If in R, reclassify is the wrong tool I think, that requires three values per column, a from, to interval and the new to assign those.

Do you want to replace -999 with NA? Then you need something like

r <- mydat[[1]]; r[r == -999] <- NA;

and then a way to write out each slice to a new data set.

See vignette("functions") for other approaches for large data.