I have two years worth of GIS data contained in CSV files stored in nested directories (per year, per month) so it looks like:
2013 01 2013-01-01.csv 2013-01-02.csv etc. 02 etc. 2014 01 2014-01-01.csv 2014-01-02.csv etc. 02 etc.
The files contain the fields: ID, timestamp, lat, long. Combined, it is around 50GB worth of data. I am looking for the most efficient way to load data stored in a folder hierarchy into a single table in PostGIS. (This is my 'training data', and I'll be working with 500GB later on.)
At the moment I am thinking about writing a PyQGIS script based on os.walk (still have to work out how, tips are very welcome) for loading all the points into QGIS, saving them into the same layer, and then loading that layer into PostGIS. But I can imagine this would not be most efficient.
Any suggestions for alternative routes?