0

I worked on this a few days and got fairly far with it but I still cannot figure out how, or why, I fail to convert the epoch time stamp to a datetime. I tried several methods but I since changed the code so that it loads all the nc files into one dataframe then does the filtering. All I need do now is add a field and populate it with a datetime from the epoch and print to csv. This is how it looks now...

    ## Imports for the tool
import os, time, datetime, xarray as xr, pandas as pd

## Time tracking
mtimestart = time.time() ## Capture the time the whole process started
n = 0 ## Tracking the iterations i.e. the number of files to be merged into the dataframe

## Spatial Extent Bounds (Lat\Lon)
lat_ll = 35
lat_ul = 60
lon_ll = -75
lon_ul = -40

## Set the directory variables
in_dir = r'somedir' ## NC File Locations
out_dir = r'some other dir' ## Out Location for outputs

## Empty dataset list
dfList = []

## Set the NC File Directory as the current working directory
os.chdir(in_dir)

## Iterate over NC Files and load into dataframes
print('Building the list of Dataframes')
for file in os.listdir(os.getcwd())[:1]:
    n = n+1 ## Tracking iterations
    print(n)
    startt = time.time()
    ds = xr.open_dataset(file)
    df = ds.to_dataframe()
    dfList.append(df)
    stopt = time.time()

## Merge the list of Dataframes into on Dataframe
print('Dataframe list completed, mergining...')
df = pd.concat(dfList)
print('netCDF files merged into dataset')

## Filter the dataframe
df = df[df['latitude'].between(lat_ll, lat_ul) & df['longitude'].between(lon_ll, lon_ul)] ## Spatial Bounds Filter
print('Data clipped to spatial bounds')

#df[df['mmsi'] == mmsi] ## MMSI Filter
print("MMSI filter applied")

## Add new column and populate with datetime stamp from epoch number
df['BaseDateTime'] = (datetime.datetime.fromtimestamp(df['date_num'], datetime.timezone.utc))
print(len(df.index))

os.chdir(out_dir)
#df.to_csv(r'AIS_Atlantic_CSV.csv')

## Time tracking
mtimestop = time.time() ## Capture the time the whole process stopped
print(df)
print('Time elapsed '+str(round(mtimestop - mtimestart))) ## Delta time for process

At line 47 where I try to convert the I get

raise TypeError(f"cannot convert the series to {converter}")
TypeError: cannot convert the series to <class 'int'>

I tried the method in another python file

import datetime

unix = 1546300813

cdate = datetime.datetime.fromtimestamp(unix, datetime.timezone.utc)
print(cdate)

and it works printing out the expected date time (note I tried datetime.datetime.fromtimestamp(unix).strftime('%Y-%m-%d %H:%M:%S') but the date was off, perhaps about 24h?), when I feed it a string it comes back with an error expecting int so I tried...

(datetime.datetime.fromtimestamp(int(df['date_num']),datetime.timezone.utc))

Which produced the error

raise TypeError(f"cannot convert the series to {converter}")
TypeError: cannot convert the series to <class 'int'>

My interpretation of

df['BaseDateTime'] = (datetime.datetime.fromtimestamp(df['date_num'], datetime.timezone.utc))

Is I am creating a new column called BaseDateTime and populating it with data from date_num that I have doing some function to, like converting to int and passing to a datetime converter.

Suggestions?

0

2 Answers 2

0

The NetCDF4 library has a num2date() function which should handle this, w/o the need for the datetime module.

# this is the NetCDF4 variable with all it's attributes, including time units.
unix_date_num_var = in_ncf_ds.variables['date_num']
# this is just the list of number values.
unix_date_num =  unix_date_num_var[:]

BaseDateTime_list = NetCDF4.num2date(unix_date_num, unix_date_num_var.units)
5
  • It throws an error "AttributeError: NetCDF: Attribute not found" on line "BaseDateTime_list = netCDF4.num2date(unix_date_num, unix_date_num_var.units)" it might not be stored as a netCDF date, it might just be stored as a number. May 6 at 13:54
  • 1
    Do you have access to ncdump, installed with the NetCDF4 C library. Run ncdump -h myData.nc to view the variable attributes including the date_num variables attributes. A valid NetCDF file should have a .units attribute. May 7 at 15:40
  • Cool, I get units listed as seconds since 1970-Jan-01 00:00:00, all my other values just say Dindex May 9 at 12:00
  • See my response at stackoverflow.com/a/72165563/1211981 May 9 at 12:05
  • I worked on this a bit more, the original question was quite broad in terms of gathering data from netcdf files, and I updated the question and my code. I since learned how to do much of it through pandas and xarray but still datetime is escaping me. Of course my new approach passes the epoch number to the datetime converter a bit differently. May 10 at 19:11
0

Well this worked

['BaseDateTime'] = pd.to_datetime(df['date_num'], unit='s', utc=True)

Not sure why the other suggestions didn't.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.