3

I have a few windows directories that each have many 1000's of Geotiff DSM/DEM files downloaded from LINZ (Land information NZ) These are derived from 1m LiDAR and each file covers a couple km2

I can import these individually, however each directory is 8GB and i have given up waiting importing by dragging over all TIF, its has been left for hours, stops responding at times etc.

I just tried building a virtual raster using one directory as input, it lists 18586 files as directory includes prj, aux, xml tfw supporting files. However I get error message "Incorrect Parameter value for INPUT"

Google has not been my friend and i cannot find a way to access these easily.

I'm looking for a solution where they get rendered as needed and access / manage 10,000's raster files across NZ using whatever is 'best practice'

Hope someone can point me in the right direction! I've been using QGIS for years and use Postgres for some vector files.

I don't think this is version related - i have tried LTS and 3.18

Improve this question
4
  • Have you built pyramids or overviews to speed up the rendering?
    – BERA
    Jun 11, 2021 at 5:11
  • If you want that a very large directory of images is fast at any scale you must have a two-way solution. For large scales when the area on screen covers a subset of one image it is enough to have vrt and overviews on each image. VRT is not fast for showing the whole area on screen because opening 10000 source images just cannot be fast. For small scales the source data must be preprocessed to have less files and larger pixels for example with gdal.org/programs/gdal_retile.html. If you do not need to show DEM at small scales just set scale limit for rendering.
    – user30184
    Jun 11, 2021 at 7:44
  • Slides about the idea behind gdal_retile - open as few files as possible slideshare.net/geosolutions/…. The same principle is valid for any software.
    – user30184
    Jun 11, 2021 at 7:50
  • Thanks you for yoru comments: I did reply last week however just noticed it didn't post! I have installed geoserver and need more time to work out how to load a directory, I'm failing with a single goetiff so far! It could be a useful option for some of these larger files, especially as i use up to 6 different QGIS Projects with similar data. An dyes the pyramiding is on the cards, i just need to be able to load and manage the data set first as it's big!
    – Melbnz
    Jun 25, 2021 at 5:01

1 Answer 1

Reset to default
0

The better is to process your files without charging it in QGIS with the processing toolbox and make pyramids and optimized raster file.

Here is the step I would advise :

  1. Have time to wait for the process to be done. Prepare the data to get the easier and quicker processing. If you have an SSD, you can use it to store your data to gain speed. You may verify you have sufficent space in the hard drive where QGIS is installed.
  2. If you have in each folder different raster format or different type of data for the same sector, you have to extract only the file you want to merge in a specific folder. Command line or copying tools could help you to do it in one step.
  3. Open QGIS, add one file to check the data could be open, and it is what you want. Then deactivate the rendering to get QGIS usable even with huge dataset.

enter image description here

  1. Open the tool to merge raster, for input layer click on ... then Add directory and choose a first directory to process. Set merged to be saved to a file in the format you want (geotiff or geopackage will be my choice) and uncheck Open output file after running algorithm.

enter image description here

  1. Open the tool build overviews and choose the file you have produced before as input and choose to produce as many pyramid stage as you think it is necessary.
  2. Add the layer in your QGIS project (without rendering), right click and Set scale visibility to 1:10000 or lower if you have metric data for the first try. Activate rendering and see if it work well (you can activate visibility at greater scale step by step).

Note : I did this process for 20 cm resolution ortho in a region of 1500 km². May be for your dataset size it would not be the best solution, or you have to split it.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.