0

I have a CSV file of my study area and I want to choose best area in this large study area to run my modelling for hydrogeology which is based on maximum event precipitation.

I have the code and I can indexing and get the value by thresholding.

For example: I choose index 1450 and show the value more than 100.

How can I identify the values that I want and get rid of the useless data, delete them and save it to new CSV file?

Or how can I just display the valuable data with dates and pointid of that value?

I want to show the value based on the date and count of how many times that event happened.

import pandas as pd
import os

data_all= pd.read_csv('D:/Project_by_tiff_SA/CSV file/GPM2points_SA.csv')

rain_all=data_all[data_all.columns[pd.Series(data_all.columns).str.startswith('D_')]]

#Edited code
n, m = rain_all.shape
threshold = 50
out_dir = 'D:/Project_by_tiff_SA/CSV file'


for i in range(m):
    foo = rain_all.iloc[:,[i]] # Saving values of particular column in foo
    thresh = foo[foo > threshold].dropna() # Get the matching values and drop NaN
    n_thresh = len(thresh)

 if n_thresh > 0:
        out_bn = thresh.columns[0] + '.csv'
        out_file = os.path.join(out_dir, out_bn)
        print('Number of instances matching creteria is %d' % n_thresh)
        thresh.to_csv(out_file, index=True)
        print(out_file)

Attachment is the test file to use for code.

2
  • Try giving output folder name without space or any special character. Any output is getting printed ? are you getting any error ?
    – S. Thiyaku
    Jun 21 at 15:57
  • I have tried this : out_dir = 'D:/Project_by_tiff_SA/test/' No error, nothing. running without error yet no output
    – Hadi
    Jun 21 at 17:04

1 Answer 1

0

There is no need to transpose the data frame, the column can be selected directly using iloc. foo = rain_all.iloc[:,[123]]

Next while selecting the values matching the threshold, it will throw NaN whenever the particular is not matching criteria in pandas data frame. NaN can be removed by using dropna function

thresh = foo[foo > 500].dropna()

Number of values matching the creteria can be found out using len

n_thresh = len(thresh)

The entire code is as follows:

import pandas as pd
import os

data_all = pd.read_csv('Test_area_CSV.csv')

rain_all = data_all[data_all.columns[pd.Series(data_all.columns).str.startswith('D_')]]


#If I index like this I'd get the NAN table 
#I need the values to be shown based on date for whole 17000 points
n, m = rain_all.shape
threshold = 50
out_dir = './' # Change the path to where you want to save
for i in range(m):
    foo = rain_all.iloc[:,[i]] # Saving values of particular column in foo
    thresh = foo[foo > threshold].dropna() # Get the matching values and drop NaN
    n_thresh = len(thresh)
    
    if n_thresh > 0:
        out_bn = thresh.columns[0] + '.csv'
        out_file = os.path.join(out_dir, out_bn)
        print('Number of instances matching creteria is %d' % n_thresh)
        thresh.to_csv(out_file, index=True)
        print(out_file)
5
  • Man thank you for your response One line of code I think did not work correctly! foo = rain_all.iloc[:,:]>50 ` #I want to iterate all rows and columns and I think this way is correct foo = rain_all.iloc[:,:]>50 ` But after running this line, when I check the result I see nothing while I have some values more than 50. After running the line of code for dropna() all values will be removed. And last part of code did not save any file, I'll be grateful if you help me with this.
    – Hadi
    Jun 21 at 13:19
  • Updated the answer with for loop and added variable to define output directory.
    – S. Thiyaku
    Jun 21 at 14:28
  • I've checked and did exactly as you said, yet I did not received any saved file!! n, m = rain_all.shape threshold = 50 out_dir = 'D:/Project_by_tiff_SA/CSV file' for i in range(m): foo = rain_all.iloc[:,[i]] thresh = foo[foo > threshold].dropna() # Get the matching values and drop NaN n_thresh = len(thresh) ` if n_thresh > 0: out_bn = thresh.columns[0] + '.csv' out_file = os.path.join(out_dir, out_bn) print('Number of instances matching creteria is %d' % n_thresh) thresh.to_csv(out_file, index=True) print(out_file)`
    – Hadi
    Jun 21 at 14:48
  • Can you edit your question and update the latest code ?
    – S. Thiyaku
    Jun 21 at 15:29
  • 1
    Yes, I did, edited the code
    – Hadi
    Jun 21 at 15:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.