1

I am wondering about how to proportionally split or subset the gridded dataset in netCDF format where data sources were provided by European Climate Assessments and dataset (link for data source). However, E-OBS climate gridded dataset were given for each 15 years climate observation throughout Europe and working with big netCDF format grid data in R always cause memory problem in my site when I tried to treat all missing values. After decompressing grid data in netCDF format (download link of grid data), I used raster::stack to import whole gridded data, then I tried to treat missing values in this grid data because I need to do some statistics on this temperature observation later on, but it caused memory problem. So here is what in R:

mydat <- raster::stack("~/data/tg_0.25deg_reg_1980-1994_v17.0.nc")
> print(mydat)
File C:\Users\jvrat\Documents\stella\data\tg_0.25deg_reg_1980-1994_v17.0.nc (NC_FORMAT_CLASSIC):

     1 variables (excluding dimension variables):
        short tg[longitude,latitude,time]   
            long_name: mean temperature
            units: Celsius
            standard_name: air_temperature
            _FillValue: -9999
            scale_factor: 0.00999999977648258

     3 dimensions:
        longitude  Size:464
            long_name: Longitude values
            units: degrees_east
            standard_name: longitude
        latitude  Size:201
            long_name: Latitude values
            units: degrees_north
            standard_name: latitude
        time  Size:5479   *** is unlimited ***
            long_name: Time in days
            units: days since 1950-01-01 00:00
            standard_name: time

    5 global attributes:
        Ensembles_ECAD: 17.0
        Conventions: CF-1.4
        References: http://www.ecad.eu\nhttp://www.ecad.eu/download/ensembles/ensembles.php\nhttp://www.ecad.eu/download/ensembles/Haylock_et_al_2008.pdf
        history: Wed Apr  4 11:17:28 2018: ncks -a -d time,10957,16435 tg_0.25deg_regular_1.nc tg_0.25deg_reg_1980-1994_v17.0.nc
Wed Apr  4 11:16:02 2018: ncks -a --mk_rec_dmn time tg_0.25deg_regular.nc tg_0.25deg_regular_1.nc
        NCO: 4.6.7

mydat <- raster::reclassify(mydat, cbind(NA, -999))
Error: cannot allocate vector of size 2.9 Gb

To avoid this memory problem, perhaps I can try to split original gridded data in netCDF format in 10 years and 5 years observation or equally split the grid data by each 5 years climate data observation (originally it was 15 years climate data observation for whole Europe). But doing this way is not feasible and desired.

UPDATE: objective:

I want to either convert this netCDF file to plain text tabular data in ASCII format (extract plain text data of Germany grid from chosen E-OBS netCDF grid data). I can use R to work with this data manipulation, then I will use its output data (plain text data in ASCII) in ArcGIS desk for different interpolation.

UPDATE 2:

I need to extract plain text data of Germany grid with 0.25 degree resolution from original E-OBS netCDF files where 15 years climate observation for whole Europe were stored in each nc file. I need all grid data in ASCII format or csv format because I can import them with ArcGIS desktop and apply respective geospatial analysis. I need to find a solution how to get this task done that I specified above. Any idea please?

How can I make this happen this extraction or possible coercion of netCDF data? Is that doable in R? Any more thoughts?

  • @mdsumner please take a look my updated post. Basically, I want to extract plain text data of Germany' grid from chosen netCDF files. Any way to get this done in R or other tools? Thank you – Dan May 2 '18 at 12:55
2

Are you going to work with these data in R? Then only do the resetting of values with the a slice at a time. See how mydat[[1]] compares to mydat.

Are you going to work with it as a NetCDF elsewhere? R's raster is not a good tool for creating a new NetCDF file or modifying one. See ncdf4 or RNetCDF package for that.

If in R, reclassify is the wrong tool I think, that requires three values per column, a from, to interval and the new to assign those.

Do you want to replace -999 with NA? Then you need something like

r <- mydat[[1]]; r[r == -999] <- NA;

and then a way to write out each slice to a new data set.

See vignette("functions") for other approaches for large data.

  • working with this data in R is to get only particular grid data, then work with its output in ArcGIS desktop. For chosen E-OBS grid data of whole Europe, I want to extract plain text data in ASCII format from netCDF file. The purpose of doing that is I intend to do yearly statistics for all grid observations of Germany. I know how to crop Germany's grid, but I need to treat missing values before the clipping.Could you elaborate your motivation please? – Dan May 2 '18 at 10:46
  • Is this only one layer? I would crop before changing the values. r <- crop(mydat, extent(<of-germany>)) r[r == -999] <- NA` - if it's more than one layer, let us know. Put the output of print(mydat) in the question please. – mdsumner May 2 '18 at 11:13
  • it is not only one layer, in total 5497 layer in rasterstack object, please take a look updated post. Any possible update on your solution? Thank you ! – Dan May 2 '18 at 14:45
  • It's all pretty straightforward in R, see ?raster::crop ?raster::subset ?raster::as.data.frame and ?raster::writeRaster - as it stands your question can't be answered without going and doing everything, it's too much to guess at - I suggest trying each of these things in the small so you learn about the tools available to you. Reclassify is the wrong function, and mydat[mydat == -999] <- NA might be enough, but do your cropping first. – mdsumner May 3 '18 at 0:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.