0

Suppose I have multiple HDF5 files in my E:\NASA directory and want to compile/concatenate the lat (path in HDF5: xxx.HDF5/XS/FLG/latitude), long (path in HDF5: xxx.HDF5/XS/FLG/longitude) and precipitation (path: HDF5: xxx.HDF5/XS/SLV/precipRateESurface) into a single CSV file in Python so that the lat is in column A, long is in column B, and precipitation is in column C when I open the CSV file in Excel.

I would also like to set the latitude to 25N 41N and longitude to 67E 103E. I need help making a logical argument so that if both latitude and longitude are the same then only precipitation data gets added else all the latitude, longitude and precipitation data is added as new rows in the CSV file. Also np.meshgrid doesn’t work as values are too large.Output is below input if required

The incomplete code[which only converts a single HDF file’s data to 3 CSV files without subsetting the latitude and longitude. I don’t want to merge the 3 types of data from a single file as they are unique ]:

import h5py as h
import numpy as np


f = h.File('E:/NASA/2A.GPM.DPR.V8-20180723.20170314-S234229-E011502.017290.V06A.HDF5', 'r')
g = f.keys()
print(g)

data = f['MS']
M = np.array(data)
print(M)

data2= f['MS/Latitude'][:]
data9 = f['NS/Latitude'][:]
data10 = f['HS/Latitude'][:]
N = np.array(data2)
AF = np.array(data9)
AG = np.array(data10)
print(N,AF,AG)

data3= f['MS/Longitude'][:]
data11 = f['NS/Longitude'][:]
data12 = f['HS/Longitude'][:]
O = np.array(data3)
AH = np.array(data11)
AI = np.array(data12)
print(O,AH,AI)

#Latitude and longitude data vary by hdf5 files
# I need help making a logical argument so that if latitude longitude are same then only precipitation data gets added else all long lat data precip is added

data4 = f['MS/SLV/precipRateESurface'][:]
data5 = f['NS/SLV/precipRateESurface'][:]
data6 = f['HS/SLV/precipRateESurface'][:]
P = np.array(data4)
AB = np.array(data5)
AC = np.array(data6)
print(P,AB,AC)



# Additional information

W = N.shape,AF.shape,AG.shape
print(W)

Y = O.shape,AH.shape,AI.shape
print(Y)


Z = P.shape,AB.shape,AC.shape
print(Z)

datax = f.get('AlgorithmRuntimeInfo')
X = np.array(datax)
print(X)

#np.meshgrid doesnt work as values are too large

output = np.column_stack((N.flatten('F'),O.flatten('F'),P.flatten('F')))
np.savetxt('E:/output8.csv',output,delimiter=',',fmt='%f')

output1 = np.column_stack((AF.flatten('F'),AH.flatten('F'),AB.flatten('F')))
np.savetxt('E:/output9.csv',output1,delimiter=',',fmt='%f')

output2 = np.column_stack((AG.flatten('F'),AI.flatten('F'),AC.flatten('F')))
np.savetxt('E:/output10.csv',output2,delimiter=',',fmt='%f')

Output:

runfile('C:/Users/admin/.spyder-py3/temp.py', wdir='C:/Users/admin/.spyder-py3')
<KeysViewHDF5 ['AlgorithmRuntimeInfo', 'HS', 'MS', 'NS']>
['ScanTime' 'scanStatus' 'navigation' 'PRE' 'VER' 'CSF' 'SRT' 'DSD'
 'Experimental' 'SLV' 'FLG' 'TRG' 'Latitude' 'Longitude']
[[-65.688995 -65.64181  -65.59476  ... -64.66737  -64.62054  -64.57319 ]
 [-65.68896  -65.64176  -65.59471  ... -64.667244 -64.6204   -64.573044]
 [-65.68879  -65.6416   -65.59455  ... -64.66711  -64.620255 -64.5729  ]
 ...
 [-65.68947  -65.64229  -65.595245 ... -64.66811  -64.62129  -64.57395 ]
 [-65.689644 -65.642456 -65.59542  ... -64.66818  -64.62135  -64.57401 ]
 [-65.68968  -65.642494 -65.59546  ... -64.66821  -64.62138  -64.57404 ]] [[-66.27515  -66.22456  -66.17403  ... -64.08867  -64.03802  -63.98727 ]
 [-66.27512  -66.22453  -66.174    ... -64.0885   -64.03784  -63.987095]
 [-66.27491  -66.22433  -66.1738   ... -64.088326 -64.03766  -63.986908]
 ...
 [-66.27552  -66.224945 -66.174416 ... -64.0896   -64.038956 -63.988224]
 [-66.27571  -66.225136 -66.17461  ... -64.08962  -64.03898  -63.988247]
 [-66.27573  -66.22514  -66.17462  ... -64.08961  -64.038956 -63.988224]] [[-65.66401  -65.61695  -65.57023  ... -64.689476 -64.64274  -64.5956  ]
 [-65.66384  -65.61678  -65.57007  ... -64.68934  -64.6426   -64.59547 ]
 [-65.663666 -65.6166   -65.569885 ... -64.689125 -64.64239  -64.595245]
 ...
 [-65.66461  -65.61755  -65.57085  ... -64.6903   -64.64358  -64.59644 ]
 [-65.66466  -65.61761  -65.5709   ... -64.69035  -64.64363  -64.596504]
 [-65.66466  -65.61761  -65.5709   ... -64.690315 -64.643585 -64.59646 ]]
[[-18.237562 -18.236645 -18.235638 ... -18.19753  -18.194767 -18.191917]
 [-18.130909 -18.130188 -18.129374 ... -18.094954 -18.09237  -18.0897  ]
 [-18.024227 -18.023706 -18.02309  ... -17.992405 -17.990004 -17.987518]
 ...
 [-41.91062  -41.90923  -41.90775  ... -41.860657 -41.857456 -41.854164]
 [-41.803974 -41.802776 -41.80149  ... -41.75807  -41.75505  -41.75194 ]
 [-41.697212 -41.696213 -41.69512  ... -41.655468 -41.652634 -41.649708]] [[-18.240753 -18.24108  -18.241306 ... -18.158884 -18.15509  -18.151217]
 [-18.131588 -18.132137 -18.132584 ... -18.058481 -18.054874 -18.051186]
 [-18.022392 -18.023163 -18.023829 ... -17.958141 -17.954721 -17.951223]
 ...
 [-41.91987  -41.91966  -41.919353 ... -41.816696 -41.812447 -41.80812 ]
 [-41.810753 -41.81076  -41.810673 ... -41.71631  -41.71225  -41.708107]
 [-41.701477 -41.70171  -41.70184  ... -41.615948 -41.61208  -41.60813 ]] [[-18.18645  -18.185621 -18.184692 ... -18.150003 -18.147316 -18.144539]
 [-18.079859 -18.079226 -18.07849  ... -18.04733  -18.044825 -18.042229]
 [-17.973377 -17.97294  -17.972399 ... -17.944761 -17.942432 -17.940018]
 ...
 [-41.859287 -41.857983 -41.856583 ... -41.813328 -41.8102   -41.806984]
 [-41.75266  -41.75155  -41.75034  ... -41.710617 -41.707672 -41.70464 ]
 [-41.646053 -41.645145 -41.644135 ... -41.608036 -41.605278 -41.60243 ]]
[[0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 ...
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]] [[0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 ...
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]] [[0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 ...
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]]
((7934, 25), (7934, 49), (7934, 24))
((7934, 25), (7934, 49), (7934, 24))
((7934, 25), (7934, 49), (7934, 24))
[b'2A.GPM.Ku.V8-20180723.20170314-S234229-E011502.017290.V06A.HDF5,2A.GPM.Ka.V8-20180723.20170314-S234229-E011502.017290.V06A.HDF5\nPRE/kuns_adjust_factor_05c_1.dat\nPRE/kams_adjust_factor_05c_1.dat\nPRE/kahs_adjust_factor_05c_1.dat\nVER/DPR-L2_VER_CLW-DB_ver20150508.dat\nCSF/w_outDPR_CSF.txt\nDSD/log10eps_5deg_v20170119_0120.dat\nDSD/log10eps_5deg_v20161113_0030.dat\nDSD/log10eps_5deg_v20170119_0120.dat\nTRG/TRG_sample_DB.dat\nTRG/TRG_sample_LUT.dat\nSRT/Temporal_0.5F_3YMAM2017_6S_V05A_UF.bin\nSLV/lut_2015_0827_wibb_nw_dm_m3.dat\nSLV/lut_2015_0827_wobb_nw_dm_m3.dat\nSLV/paramset_2017_0129a_idm5000_ip200_ic8.dat\nSLV/paramset_2017_0129a_ip200_ic8.dat\nSLV/DB_for_esurf2_20161017.dat\nSLV/paramset_2017_0129a_ip200_ic8.dat']
  • Can you clarify your desired output? Currently you get 3 .csvs with columns for lat, lon, and precip ... Do you want those same 3 outputs but leave the lat/lon values blank if they are equal to 25N 41N, 67E 103E? – Nick Jun 18 at 15:31
  • No I want to leave the whole lat long precip blank if they are not BETWEEN the maximum and minimum N, E coordinates for eg. if data is 40N, 68E=2; 40N, 105E(exceeds103E)=3 then only data of 40N,68E=2 gets added. Also the other thing I want is for example in file 1 there is only 1 data like 0N,1E = 25.55;0N, 2W= 10 and in file 2 the data is 0N,1E = 5.55; 0N,1W=10.55 then i want output 0N, 1E = 25.55 + 5.55 but the 0N, 2W = 10 and 0N, 1E = 5.55 don't get added to each other but are added to csv file as separate values – Weathercoding Jun 19 at 15:59
  • @Jose answered the first part about filtering by lat / lon. For the second part, building of @Jose's answer, you could add the line df = df.groupby(['lat', 'lon']).sum() immediately before exporting to csv – Nick Jun 19 at 16:13
  • Excuse me for being that direct but how this question is related to GIS SE, IMHO it is more a pure SE stuff, is not it? – Taras Jun 19 at 19:52
  • Taras when I posted an earlier question on data science it did not get any attention but when I posted it GIS I got adequate attention so I decided this question in GIS too – Weathercoding Jun 20 at 12:48
1

I'm not sure what your problem is running Jose's code it: seems to work fine for me (except that I had to change ds.varaibles to simply ds when getting lat/lon/precip, but that may be because I'm using slightly different GPM files). I'm guessing something is wrong with the values you are passing for min_lon, max_lon etc ... but hard to say

Here is something that builds off their method but using some of the code you already have. Instead of filtering the data frames inside the function it just filters them after they are all in one data frame, and then uses pd.DataFrame.groupby() to sum up the precip values for those records with matching coordinates. It also iterates over all three sensor types included in your original example.

from pathlib import Path

import h5py as h
import numpy as np
import pandas as pd

def extractLatLonPrecip(file, scan_type):
    f = h.File(file, 'r')
    
    lat_data = f['%s/Latitude' % scan_type][:]
    lon_data = f['%s/Longitude' % scan_type][:]
    precip_data = f['%s/SLV/precipRateESurface' % scan_type][:]
    
    lat_ndarray = np.array(lat_data)
    lon_ndarray = np.array(lon_data)
    precip_ndarray = np.array(precip_data)

    output_ndarray = np.column_stack((lat_ndarray.flatten('F'),lon_ndarray.flatten('F'),precip_ndarray.flatten('F')))
    output_df = pd.DataFrame(output_ndarray, columns=['lat','lon','precip'])
    return output_df
    

hdf5_directory  = Path(r"E:/NASA/")
file_list = sorted([f for f in hdf5_directory.glob("2A.GPM*HDF5")])

min_lat,max_lat = 25,41
min_lon,max_lon = 67,103

scan_type_list = ['HS','MS','NS']

for scan_type in scan_type_list: 
    
    df = pd.concat([extractLatLonPrecip(file, scan_type) for file in file_list])
    
    df = df[df.lat >= min_lat]
    df = df[df.lat <= max_lat]
    df = df[df.lon >= min_lon]
    df = df[df.lon <= max_lon]

    df = df.groupby(['lat', 'lon'], as_index=False).sum()
    
    df.to_csv(Path(hdf5_directory.parent,"%s.csv" % scan_type), index=False)

| improve this answer | |
  • I cannot find the output in my computer although python says it has sent the file somewhere .How I can I specify the location of file to be saved in with the string formatter? – Weathercoding Jun 20 at 13:53
  • Currently each csv file is getting saved to the directory above the directory that contains the hdf5 files. So if you haven't changed it all the are all in E:\. – Nick Jun 20 at 14:00
  • df.to_csv(Path(hdf5_directory,"%s.csv" % scan_type), index=False) to save in the same directory as the hdf5s, or df.to_csv("E:/PATH/TO/OUTPUT/LOCATION/%s.csv" % scan_type, index=False) to specify path explicitly – Nick Jun 20 at 14:05
  • Sorry to irritate you but the lat long coordinates are not in the csv file. Please can u share how can I add the filtered lat long coordinates alongside the precipitation values in the csv file – Weathercoding Jun 20 at 16:45
  • 1
    try df = df.groupby(['lat', 'lon'], as_index=False).sum() – Nick Jun 21 at 17:03
1

Your question is quite convoluted and conflates two different problems. Let's see... You want to create a CSV file with lat, lon and precip_rate as columns from a bunch of files, and only covering a particular spatial region. Let's just do this for one of your files, then extend to all of them.

From your question, you need the Latitude, Longitude and PRE (?) variables

import h5py
import numpy as np
import pandas as pd
def extract_precip(fname, min_lon, min_lat, max_lon, max_lat,
                  field="PRE"):
    ds = h5py.File(fname)
    lat = ds['MS/Latitude'][:]
    lon = ds['MS/Longitude'][:]
    passer1 = np.logical_and(lat >= min_lat, lat <= max_lat)
    passer2 = np.logical_and(lon >= min_lon, lon <= max_lon)
    passer = passer1 * passer2
    precip = ds['MS/SLV/precipRateESurface'][:]
    # You should check whether passer.sum() > 0
    # in that case, return `None`
    if passer.sum() == 0: # No valid samples
        return None
    df = pd.DataFrame({'lat':lat[passer],
                       'lon':lon[passer],
                       'pre':precip[passer]
                       # You may also want to store the date here 
                       # or something else
                      })
   return df

The previous snippet should work, if I got the variable names within the HDF5 files right (not sure, it's all jumbled in your code, but you get the gist). Now, to apply this a bunch of files, you'd need to open them, and stick together all the data. The next snippet does that using the function above

from pathlib import Path

loc = Path("E:/NASA/")

files = sorted([f for f in loc.glob("2A.GPM*HDF5")])
print(f"Found {len(files)} hdf5 files that match the pattern")

df = pd.concat([extract_precip(
                fixx, min_lon, min_lat, max_lon, max_lat)
                for fixx in files] )
df.to_csv("super_funky_file.csv", index=False)

So that would allow you to do what I think you say you want. I think you may want to extract other fields too (e.g. time or something), but it's quite clear how you go about it.

| improve this answer | |
  • Also the other thing I want is for example in file 1 there is only 1 data like 0N,1E = 25.55;0N, 2W= 10 and in file 2 the data is 0N,1E = 5.55; 0N,1W=10.55 then i want output 0N, 1E = 25.55 + 5.55 but the 0N, 2W = 10 and 0N, 1E = 5.55 don't get added to each other but are added to csv file as separate values – Weathercoding Jun 19 at 16:25
  • I don't understand what you want here, but can't you just process the spatial filter in the DataFrame? – Jose Jun 22 at 16:11
  • I don't know much about pandas – Weathercoding Jun 23 at 8:59
  • Check the logic in the passer1 and passer2 and tailor it to your needs. – Jose Jun 23 at 13:01
  • You may want to turn this into another question and add a bit more detail and examples. – Jose Jun 24 at 13:50

Not the answer you're looking for? Browse other questions tagged or ask your own question.