0

My final objective is to perform a regression (latitude predicted by precipitation).

To achieve that, I have download this dataset from Copernicus: https://surfobs.climate.copernicus.eu/dataaccess/access_eobs.php#datafiles

Here, I have faced my first challenge as I have never used NetCDF data source before. Following some good advice found on StackExchange, I have followed this author guidelines: https://semba-blog.netlify.app/11/03/2018/converting-netcdf-files-into-data-frame/ where he shows how to: Read and Convert NetCDF files into data frame in R.

I have managed to reproduce pretty well everything explained, I think, but I am missing something: I have a 3-dimensions variable, as precipitation (rr) is function of latitude, longitude and time:

> print(rr)
File C:\Users\Dell\Desktop\~\climate_data\rr_ens_mean_0.1deg_reg_v25.0e.nc (NC_FORMAT_NETCDF4):

     1 variables (excluding dimension variables):
        short rr[longitude,latitude,time]   (Chunking: [705,465,1])  (Compression: level 1)
            units: mm
            _FillValue: -9999
            long_name: rainfall
            scale_factor: 0.100000001490116
            add_offset: 0
            standard_name: thickness_of_rainfall_amount
            cell_methods: time: mean

     3 dimensions:
        longitude  Size:705 
            units: degrees_east
            long_name: Longitude values
            axis: X
            standard_name: longitude
        latitude  Size:465 
            units: degrees_north
            long_name: Latitude values
            axis: Y
            standard_name: latitude
        time  Size:26298   *** is unlimited *** 
            units: days since 1950-01-01 00:00
            long_name: Time in days
            calendar: standard
            standard_name: time
            cell_methods: time: mean 

However, in the process followed I am "losing" the time information. Basically, here is how my data frame looks:

Random sample of observations showing the rainfall
Longitude   Latitude    rainfall
-8.9501396  50.94986    NA
21.5498603  49.54986    0.0
16.0498603  70.74986    NA
-18.7501395 45.94986    NA
13.7498603  61.14986    3.9
13.5498603  31.94986    0.0
19.0498603  29.94986    NA
32.3498603  30.94986    0.0
-24.1501395 31.64986    NA
33.7498603  40.74986    5.0
42.0498602  57.54986    0.0
-6.0501396  64.34986    NA
0.2498604   34.04986    1.6
20.2498603  49.04986    0.0

My code is the following:

library(ncdf4)
library(RNetCDF)
library(sf)
library(tidyverse)
library(raster)
library(kableExtra)
library(ggplot2)
library(spData)
library(oce)

### https://semba-blog.netlify.app/11/03/2018/converting-netcdf-files-into-data-frame/
  
#read the nc file
rr = raster("C:/Users/Dell/Desktop/~/climate_data/rr_ens_mean_0.1deg_reg_v25.0e.nc",
            level = 1, 
             varname = "rr")

#define projection
proj4string(rr)=CRS("+init=EPSG:4326")

#convert to data frame
rr.df = raster::as.data.frame(rr, xy = TRUE)

#random sample of 12 observations 
rr.df %>% sample_n(50) %>% kableExtra::kable("html", row.names = FALSE, col.names = c("Longitude", "Latitude", "rainfall"), align = "c", caption = "Random sample of 50 observations showing the Sea level pressure") %>%
  kableExtra::column_spec(column = 1:3, width = "5cm", color = 1)

#visualize

ggplot()+
  geom_raster(data = rr.df,  aes(x = x, y = y, fill = rainfall), interpolate = FALSE)+
  #geom_sf(data = spData::world, fill = rainfall, col = "black")+
  # geom_path(data = data, aes(x = lon, y = lat, col = id), size = .75)+
  coord_sf(xlim = c(10, 25), ylim = c(55, 70))+
  # scale_color_jco(name = "Argo float")+
  theme_bw()+
  theme(legend.position = c(.85,.2), 
        legend.background = element_rect(colour = 1),
        legend.key.width = unit(.75, "lines"), 
        legend.text = element_text(size = 11, colour = 1),
        axis.text = element_text(colour = 1, size = 12))+  
  geom_label(aes(x = 60, y = 0, label = "a"))+
  labs(x = NULL, y = NULL)+
  scale_fill_gradientn(name = "rr", colours = oce::oceColors9A(120))+
  scale_x_continuous(breaks = seq(10, 25, 5))+
  scale_y_continuous(breaks = seq(55, 70, 5))

What can I do to translate my NetCDF file into a data frame while keeping, for the precipitation, the latitude, longitude and time information that is crucial to perform my regression?

4
  • try tidync : tidync(file) |> hyper_tibble() still have to convert the raw numbers to datetime with the metadata
    – mdsumner
    Jul 9, 2022 at 14:28
  • thank you for helping! However, while doing this I have a RAM size issue (even with 80GO), by any chance would you know a way to subset my data to what is of interest before doing your line of code?
    – Recology
    Jul 10, 2022 at 10:53
  • Seems that I have find out, using the hyper_filter function! just need to find out how to convert raw numbers to datetime
    – Recology
    Jul 10, 2022 at 11:03
  • see hyper_filter for lazy subset, hyper_array and hyper_tibble materialize the data in memory after subset
    – mdsumner
    Jul 11, 2022 at 3:18

1 Answer 1

0

new user and I don't have the rep points to comment so sorry that this is only a partial answer. I've found this article: https://rpubs.com/boyerag/297592 and this should get you started:

nc_data<-nc_open("C:\Users\Dell\Desktop\~\climate_data\rr_ens_mean_0.1deg_reg_v25.0e.nc")

rr_lon <- ncvar_get(nc_data, "longitude")
rr_lat <- ncvar_get(nc_data, "latitude")
rr_time <- ncvar_get(nc_data, "time")
rr_rain <- ncvar_get(nc_data, "rr")

However, these could not be put in to a dataframe due to unequal numbers and the "NDVI" mentioned in the linked article does not seem relevant to the rainfall files but I hope this is a step in the right direction

4
  • The problem of this is that my file is too big to use a "ncvar_get" (and I have 80GO RAM). I have done some subset to try again, but then I have some error messages saying that : first argument (nc) is not of class ncdf4!
    – Recology
    Jul 11, 2022 at 8:45
  • Can I ask how you are subsetting. Is it using subset() or along the lines of this advice?: gis.stackexchange.com/questions/360225/…
    – Mark O
    Jul 11, 2022 at 23:06
  • I used different methods, including the one on your link, without success
    – Recology
    Jul 12, 2022 at 8:37
  • have you experimented with using the smaller chunks: surfobs.climate.copernicus.eu/dataaccess/access_eobs_chunks.php ? Other than that I can only suggest getting access to a more powerful computer, perhaps an R Server instance
    – Mark O
    Jul 12, 2022 at 23:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.