I downloaded gridded meteorological data as .csv file from the JRC CGMS database for a territory of interest. The file has data for predefined point locations over the territory in a regular-spacing grid. The data are daily and cover a period, say 1 year. So, in the .csv file there is a separate row for each combination of point and date (for example, we have 365 rows for the same point, each with data for particular date; then the next 365 rows for the next point. etc). The data also include geographic coordinates of the points.

I would like to automatically handle such data with the final goal of obtaining, let say, 365 shapefiles, each containing meteorological attributes from the .csv for a single date for all points.

Sample of the data (1 month; 2 points only) can be obtained from this link: https://www.dropbox.com/s/0v5764swgewsdrd/Gridded_AgroMet_Data_Europe_ver2_0_sample2.csv?dl=0

What I tried so far is to open the .csv file in QGIS and export as a point shapefile. This shapefile, however has, for each point, 365 features lying on top of each other. I want a python script to select features based on the 'DAY' attribute end export in new shapefile. And I want this procedure to iterate over all unique values of the attribute 'DAY'. So far my script is just that:

shapefile = iface.activeLayer()
selection= shapefile.getFeatures(QgsFeatureRequest().setFilterExpression("\"DAY\"='20150101'"))
shapefile.selectByIds([s.id() for s in selection])

How to continue from here? How to export the selection? How to iterate between dates?

  • Welcome to GIS SE. Would you be willing to use gdal instead of PyQGIS? – Marcelo Villa-Piñeros Jun 14 '19 at 14:25
  • Do u absolutely want to achieve this programmatically or do u accept any other method ? ;-p – Snaileater Jun 14 '19 at 16:52
  • Welcome! you might want to have a look at the NetCDF file format, It seems better suited for your needs – xlDias Jun 14 '19 at 19:05
  • Thanks for all the comments. I do not mind to use gdal if this is a convenient way, but I am unfamiliar with gdal as well as with PyQGIS. In all cases I should start from the beginning. I can do this manually but it will take several days and errors are very probable. I would prefer an automated way. The JRC site provides the data only in CSV file format. Should I convert them in NetCDF myself? Is this possible? What software should I use? – ABC Jun 17 '19 at 6:11
  • Geopandas and pandas would be suitable for this. I can post an answer if you are ok with using these libraries – BERA Jun 18 '19 at 13:17

You open the .csv file ("add delimited text layer" - CRS : WGS 84 - X and Y being the longitude and latitude attributes).

You can save it as a shape file or any format if necessary. In the Processing Toolbox you choose "split vector layer", as "unique ID Field" you choose "DAY" and the toolbox generates the awaited files ...

You're done ...

Being a toolbox treatement it should not be too hard to automate this (some posts describe the procedure)...

(not sure about the NetCDF proposal ... i don't think it will make your life easier for what u seem to be awaiting ...)

  • I did not know about this "split vector layer" tool. It did exactly what I wanted, and so easily. Thanks! – ABC Jun 20 '19 at 6:44

You can use pandas and geopandas libraries. Install these in your existing qgis python environment or install conda (and then conda install geopandas).

import geopandas as gpd
import pandas as pd
from shapely.geometry import Point
import os

csvfile = '/home/bera/Downloads/Gridded_AgroMet_Data_Europe_ver2_0_sample2.csv'
out_shapefile_folder = '/home/bera/Downloads/Outshapes/'

df = pd.read_csv(csvfile, delimiter=';') #Create pandas dataframe from csv

#Create geodataframe from dataframe
geometry = [Point(xy) for xy in zip(df.LONGITUDE, df.LATITUDE)]
crs = {'init': 'epsg:4326'} #4326 correct?
gdf = gpd.GeoDataFrame(df, crs=crs, geometry=geometry)

#Iterate over each day and export shapefile
for day, group in gdf.groupby(by='DAY'):

enter image description here

I dont know what your next step is but when the data is in pandas/geopandas you can do alot of analysis. Might be easier than handling all output shapefiles.

  • 1
    This seems to be very smart and elegant solution as well. Thanks. You are right. I will have to find a way to handle all this shp files now that I have them. I need to create raster files from the point vectors and maybe to linearly interpolate between points, i.e. to have a raster with resolution (somewhat) higher than the spacing of the original points. But this is another story. – ABC Jun 20 '19 at 7:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.