Processing a large number of raster files in QGIS

Question

I have a large number (100,000+) of raster files I need to process in QGIS (exporting to tiff, clipping to polygon layers, etc), for a total of approx 500GB. Can anyone advise if there's some way of handling these within QGIS (a raster GDB?)? Can to stretch into code if needs be, but would prefer a simple solution using embedded tools / plugins if possible. I've used batch processing tools but some of the larger sub folders have 9000 files and I'm having problems loading them into the batch processing window.

Currently height (DSM) .asc files, UK coverage but patchy. Would prefer to work within QGIS as I'm more familiar with it, but would consider any open source / free toolset.

Answer 1

You should avoid using QGIS GUI when doing this kind of operation, it tends to hang. Rather have a look to the gdal commandline family (you can access them through the osgeo4w shell) : http://www.gdal.org/gdal_utilities.html Example here : How to run OSgo4W shell script on Windows?

You might have a good use of VRT files with your data. See gdalbuildvrt for making one of them : "One vrt will rule them all" ;)

Answer 2

Handling such a humongous data set does not seem to be a good idea in QGIS and so is handling it in the dedicated plugin provided in the very package. I would suggest that you use the accessibility of GDAL in GRASS package that provides a good number of tools to deal with raster data sets with quite comprehensive modules such as r.in.gdal or r.mapcalc and do use the command line interface for it since visualization of such large data might cause your project to crash.