Loading and managing the directory of 6500 raster DSM GeoTIFF files in QGIS

Question

I have a few Windows directories that each have many 1000 of GeoTIFF DSM/DEM files downloaded from LINZ (Land Information NZ) These are derived from 1m LiDAR and each file covers a couple of km2

I can import these individually, however, each directory is 8GB and I have given up waiting to import by dragging over all TIF, it has been left for hours, stops responding at times etc.

I just tried building a virtual raster using one directory as input, it lists 18586 files as a directory including prj, aux, xml tfw supporting files. However, I get the error message "Incorrect Parameter value for INPUT"

Google has not been my friend and I cannot find a way to access these easily.

I'm looking for a solution where they get rendered as needed and access / manage 10,000 raster files across NZ using whatever is 'best practice'.

I don't think this is version-related - I have tried LTS and 3.18

Answer 1

The better is to process your files without charging it in QGIS with the processing toolbox and make pyramids and optimized raster file.

Here is the step I would advise :

0. Have time to wait for the process to be done. Prepare the data to get the easier and quicker processing. If you have an SSD, you can use it to store your data to gain speed. You may verify you have sufficent space in the hard drive where QGIS is installed.
1. If you have in each folder different raster format or different type of data for the same sector, you have to extract only the file you want to merge in a specific folder. Command line or copying tools could help you to do it in one step.
2. Open QGIS, add one file to check the data could be open, and it is what you want. Then deactivate the rendering to get QGIS usable even with huge dataset.

3. Open the tool to merge raster, for input layer click on ... then Add directory and choose a first directory to process. Set merged to be saved to a file in the format you want (geotiff or geopackage will be my choice) and uncheck Open output file after running algorithm.

4. Open the tool build overviews and choose the file you have produced before as input and choose to produce as many pyramid stage as you think it is necessary.
5. Add the layer in your QGIS project (without rendering), right click and Set scale visibility to 1:10000 or lower if you have metric data for the first try. Activate rendering and see if it work well (you can activate visibility at greater scale step by step).

Note : I did this process for 20 cm resolution ortho in a region of 1500 km². May be for your dataset size it would not be the best solution, or you have to split it.