Transforming time series gridded meteorological data from .csv file to multiple .shp files (one per date) with pyQGIS

Question

I downloaded gridded meteorological data as .csv file from the JRC CGMS database for a territory of interest. The file has data for predefined point locations over the territory in a regular-spacing grid. The data are daily and cover a period, say 1 year. So, in the .csv file there is a separate row for each combination of point and date (for example, we have 365 rows for the same point, each with data for particular date; then the next 365 rows for the next point. etc). The data also include geographic coordinates of the points.

I would like to automatically handle such data with the final goal of obtaining, let say, 365 shapefiles, each containing meteorological attributes from the .csv for a single date for all points.

Sample of the data (1 month; 2 points only) can be obtained from this link: https://www.dropbox.com/s/0v5764swgewsdrd/Gridded_AgroMet_Data_Europe_ver2_0_sample2.csv?dl=0

What I tried so far is to open the .csv file in QGIS and export as a point shapefile. This shapefile, however has, for each point, 365 features lying on top of each other. I want a python script to select features based on the 'DAY' attribute end export in new shapefile. And I want this procedure to iterate over all unique values of the attribute 'DAY'. So far my script is just that:

shapefile = iface.activeLayer()
selection= shapefile.getFeatures(QgsFeatureRequest().setFilterExpression("\"DAY\"='20150101'"))
shapefile.selectByIds([s.id() for s in selection])

How to continue from here? How to export the selection? How to iterate between dates?

Answer 1

You open the .csv file ("add delimited text layer" - CRS : WGS 84 - X and Y being the longitude and latitude attributes).

You can save it as a shape file or any format if necessary. In the Processing Toolbox you choose "split vector layer", as "unique ID Field" you choose "DAY" and the toolbox generates the awaited files ...

You're done ...

Being a toolbox treatement it should not be too hard to automate this (some posts describe the procedure)...

(not sure about the NetCDF proposal ... i don't think it will make your life easier for what u seem to be awaiting ...)

Answer 2

You can use pandas and geopandas libraries. Install these in your existing qgis python environment or install conda (and then conda install geopandas).

import geopandas as gpd
import pandas as pd
from shapely.geometry import Point
import os

csvfile = '/home/bera/Downloads/Gridded_AgroMet_Data_Europe_ver2_0_sample2.csv'
out_shapefile_folder = '/home/bera/Downloads/Outshapes/'

df = pd.read_csv(csvfile, delimiter=';') #Create pandas dataframe from csv

#Create geodataframe from dataframe
geometry = [Point(xy) for xy in zip(df.LONGITUDE, df.LATITUDE)]
crs = {'init': 'epsg:4326'} #4326 correct?
gdf = gpd.GeoDataFrame(df, crs=crs, geometry=geometry)

#Iterate over each day and export shapefile
for day, group in gdf.groupby(by='DAY'):
    group.to_file(os.path.join(out_shapefile_folder,'Raindata_date_{0}.shp'.format(day)))

I dont know what your next step is but when the data is in pandas/geopandas you can do alot of analysis. Might be easier than handling all output shapefiles.