0

I have a large number (100,000+) of raster files I need to process in QGIS (exporting to tiff, clipping to polygon layers, etc), for a total of approx 500GB. Can anyone advise if there's some way of handling these within QGIS (a raster GDB?)? Can to stretch into code if needs be, but would prefer a simple solution using embedded tools / plugins if possible. I've used batch processing tools but some of the larger sub folders have 9000 files and I'm having problems loading them into the batch processing window.

Currently height (DSM) .asc files, UK coverage but patchy. Would prefer to work within QGIS as I'm more familiar with it, but would consider any open source / free toolset.

  • Could you provide some more info's for the nature of the raster files. Are they geotiffs for example? Building the sum of the files a large mosaic (geotif - blocks vs.stripes) or have the a patchy nature. What kind of enviroment you want to work with? – huckfinn Sep 24 '17 at 8:40
  • huckfinn - see edit above. – user77875 Sep 24 '17 at 11:26
2

You should avoid using QGIS GUI when doing this kind of operation, it tends to hang. Rather have a look to the gdal commandline family (you can access them through the osgeo4w shell) : http://www.gdal.org/gdal_utilities.html Example here : How to run OSgo4W shell script on Windows?

You might have a good use of VRT files with your data. See gdalbuildvrt for making one of them : "One vrt will rule them all" ;)

2

Handling such a humongous data set does not seem to be a good idea in QGIS and so is handling it in the dedicated plugin provided in the very package. I would suggest that you use the accessibility of GDAL in GRASS package that provides a good number of tools to deal with raster data sets with quite comprehensive modules such as r.in.gdal or r.mapcalc and do use the command line interface for it since visualization of such large data might cause your project to crash.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.