Identifying the certain values and counting the event of maximum precipitation with it's date and pointID with Python

Question

I have a CSV file of my study area and I want to choose best area in this large study area to run my modelling for hydrogeology which is based on maximum event precipitation.

I have the code and I can indexing and get the value by thresholding.

For example: I choose index 1450 and show the value more than 100.

How can I identify the values that I want and get rid of the useless data, delete them and save it to new CSV file?

Or how can I just display the valuable data with dates and pointid of that value?

I want to show the value based on the date and count of how many times that event happened.

import pandas as pd
import os

data_all= pd.read_csv('D:/Project_by_tiff_SA/CSV file/GPM2points_SA.csv')

rain_all=data_all[data_all.columns[pd.Series(data_all.columns).str.startswith('D_')]]

#Edited code
n, m = rain_all.shape
threshold = 50
out_dir = 'D:/Project_by_tiff_SA/CSV file'


for i in range(m):
    foo = rain_all.iloc[:,[i]] # Saving values of particular column in foo
    thresh = foo[foo > threshold].dropna() # Get the matching values and drop NaN
    n_thresh = len(thresh)

 if n_thresh > 0:
        out_bn = thresh.columns[0] + '.csv'
        out_file = os.path.join(out_dir, out_bn)
        print('Number of instances matching creteria is %d' % n_thresh)
        thresh.to_csv(out_file, index=True)
        print(out_file)

Attachment is the test file to use for code.

Answer 1

There is no need to transpose the data frame, the column can be selected directly using iloc. foo = rain_all.iloc[:,[123]]

Next while selecting the values matching the threshold, it will throw NaN whenever the particular is not matching criteria in pandas data frame. NaN can be removed by using dropna function

thresh = foo[foo > 500].dropna()

Number of values matching the creteria can be found out using len

n_thresh = len(thresh)

The entire code is as follows:

import pandas as pd
import os

data_all = pd.read_csv('Test_area_CSV.csv')

rain_all = data_all[data_all.columns[pd.Series(data_all.columns).str.startswith('D_')]]


#If I index like this I'd get the NAN table 
#I need the values to be shown based on date for whole 17000 points
n, m = rain_all.shape
threshold = 50
out_dir = './' # Change the path to where you want to save
for i in range(m):
    foo = rain_all.iloc[:,[i]] # Saving values of particular column in foo
    thresh = foo[foo > threshold].dropna() # Get the matching values and drop NaN
    n_thresh = len(thresh)
    
    if n_thresh > 0:
        out_bn = thresh.columns[0] + '.csv'
        out_file = os.path.join(out_dir, out_bn)
        print('Number of instances matching creteria is %d' % n_thresh)
        thresh.to_csv(out_file, index=True)
        print(out_file)