0

I have NCEP/NCAR reanalysis data in netcdf format. I want to convert it into csv file. I have a script to convert netcdf to csv. But since the data in netcdf file is too large which exceeds number columns in csv file. How can i split the data in nc file to multiple sheets of csv file.

Script that i have used to convert netcdf file to csv is as follows:

library(ncdf4)
getwd()
workdir <- "I:\\NCEP\\"
setwd(workdir)
ncin <- nc_open("X27.251.228.22.106.3.5.59.nc")
print("ncin")
dname <- "hgt"
lon <- ncvar_get(ncin, "lon")
nlon <- dim(lon)
head(lon)
lat <- ncvar_get(ncin, "lat", verbose = F)
nlat <- dim(lat)
head(lat)
print(c(nlon, nlat))
t <- ncvar_get(ncin, "time")
tunits <- ncatt_get(ncin, "time", "units")
nt <- dim(t)
tmp.array <- ncvar_get(ncin, dname)
dlname <- ncatt_get(ncin, dname, "long_name")
dunits <- ncatt_get(ncin, dname, "units")
fillvalue <- ncatt_get(ncin, dname, "_FillValue")
dim(tmp.array)
title <- ncatt_get(ncin, 0, "title")
institution <- ncatt_get(ncin, 0, "institution")
datasource <- ncatt_get(ncin, 0, "source")
references <- ncatt_get(ncin, 0, "references")
history <- ncatt_get(ncin, 0, "history")
Conventions <- ncatt_get(ncin, 0, "Conventions")
# split the time units string into fields
tustr <- strsplit(tunits$value, " ")
tdstr <- strsplit(unlist(tustr)[3], "-")
tmonth = as.integer(unlist(tdstr)[2])
tday = as.integer(unlist(tdstr)[3])
tyear = as.integer(unlist(tdstr)[1])
chron::chron(t, origin = c(tmonth, tday, tyear))
tmp.array[tmp.array == fillvalue$value] <- NA
length(na.omit(as.vector(tmp.array[, , 1])))
m <- 1
tmp.slice <- tmp.array[, , m]
lonlat <- expand.grid(lon, lat)
tmp.vec <- as.vector(tmp.slice)
length(tmp.vec)
tmp.df01 <- data.frame(cbind(lonlat, tmp.vec))
names(tmp.df01) <- c("lon", "lat", paste(dname, as.character(m), sep = "_"))
head(na.omit(tmp.df01), 20)
csvfile <- "cru_tmp_1.csv"
write.table(na.omit(tmp.df01), csvfile, row.names = FALSE, sep = ",")
tmp.vec.long <- as.vector(tmp.array)
length(tmp.vec.long)
tmp.mat <- matrix(tmp.vec.long, nrow = nlon * nlat, ncol = nt)
dim(tmp.mat)
head(na.omit(tmp.mat))
# create a dataframe
lonlat <- expand.grid(lon, lat)
tmp.df02 <- data.frame(cbind(lonlat, tmp.mat))
options(width = 110)
head(na.omit(tmp.df02, 20))
csvfile <- "cru_tmp_2.csv"
write.table(na.omit(tmp.df02), csvfile, row.names = FALSE, sep = ",")
  • Since it is an ASCII flat file, there are no "maximum number of columns" in a csv file. The limitation is in the software and not the format. I would highly recommend not thinking of this data in terms of a spread sheet and articulate your analysis in a way that makes use of the format at hand. There are numerous approaches to doing this in R and your ability to perform data summaries or apply a statistical model will be notably improved. Much of this type of analysis, you will quickly find, is not tractable in a spread sheet type program. – Jeffrey Evans Nov 26 '18 at 18:04
1

Use something like this, with the high-level tools in the raster package:

library(raster)
r <- brick("X27.251.228.22.106.3.5.59.nc", varname = "hgt")
r

How does that look? If all seems well, try a plot(r[[1]]) and make sure everything's interpreted correctly, then test

tab <- as.data.frame(r[[1]], xy = TRUE)

Then if all is well, consider taking out the 1-st index subsetting there. See ?raster::as.data.frame for long and wide shape options. Once in this data.frame/tab form you can easily write to tabular formats.

The fact is that this format is so general, and the way that conventions within that format get used is so open that you almost always will need to do some upfront investigation and massaging. But once you've done that everything can be really simple and efficient.

I highly recommend using high level facilities like raster, but the upfront cost is in ensuring that the interpretation is correct. There are (some hidden) facilities in raster to deal with almost any 2D+ variable.

  • I am not able to do "r <- brick("X27.251.228.22.106.3.5.59.nc", varname = "hgt")" – user85517 May 3 '17 at 7:23
  • It shows error message " Error in .rasterObjectFromCDF(x, type = objecttype, band = band, ...) : cells are not equally spaced; you should extract values as points" – user85517 May 3 '17 at 7:24
  • Ok that requires a different approach, can you point to one file? I'll do an example – mdsumner May 3 '17 at 7:35
  • 1
    There is an undocumented argument to the brick() function stopIfNotEqualSpaced=FALSE that will allow you to open it. – Jacob F Oct 27 '17 at 19:33
  • You could use r <- as(brick("X27.251.228.22.106.3.5.59.nc", varname = "hgt", stopIfNotEqualSpaced=FALSE), "SpatialPointsDataFrame") and then just pull the @data slot to get the data.frame. – Jeffrey Evans Nov 26 '18 at 18:07
0

Here I am providing a dummy code. Note that I have used the "ncdf" package in R. You can use "ncdf4" package with a slight change of names of commands.

library(ncdf)
nc <- open.ncdf("foo.nc")             #open ncdf file and read variables
lon <- get.var.ncdf(nc, "lon")         # Lon lat and Time
lat <- get.var.ncdf(nc, "lat")
time <- get.var.ncdf(nc, "time")
dname <- "t"                         # name of variable which can be found by using print(nc)
nlon <- dim(lon)
nlat<- dim(lat)
nt<- dim(time)
lonlat <- expand.grid(lon, lat)    # make grid of given longitude and latitude 
mintemp.array <- get.var.ncdf(nc, dname)
dlname <- att.get.ncdf(nc, dname, "long_name") 
dunits <- att.get.ncdf(nc, dname, "units") 
fillvalue <- att.get.ncdf(nc, dname, "_FillValue") 
mintemp.vec.long <- as.vector(mintemp.array) 
mintemp.mat <- matrix(mintemp.vec.long, nrow = nlon * nlat, ncol = nt)
mintemp.df <- data.frame(cbind(lonlat, mintemp.mat)) options(width = 110) 
write.csv(mintemp.df, "mintemp_my.csv")

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.