I have, for a while, been trying to make a terrain map using NASA's SRTM 3 dataset. I have a large (13 GB), geotiff projected to web mercator of all the elevation files provided. I have been able to make a color-relief Geotiff based on this one. The color-relief file was made in ~1 hour.

The problem I have is that the when I try to use the hillshading it is painfully slow, and produces around 50MB an hour, on a file that I suspect will be around 20GB. (which will take 10 days+)

I have come to a point where I might just let it run in the background for the time needed, but since we have power-outages somewhat frequently from people overloading the stove, this doesn't really seem feasible!

The command I am using is

os.system("gdaldem hillshade -compute_edges  -combined -of GTiff -co BIGTIFF=YES -co compress=lzw -co predictor=2 -co TILED=YES -z 3 D:/gdal/rebase.tif D:/gdal/hill_projected.tif")

I have tried setting --config CACHE_MAX and wm, but they don't seem to bite, and the gdaldem process is still only using ~2% of the availabe memory.

I have also tried setting --config GTIFF_NUM_THREADS=ALL_CPUS which didn't speed up the task either.

I have tried to split the hillshading up to different regions, to later combine them to export to tiles, but run into the same problem in the eurasia and islands region.

  • -wm is used for reprojection algorithm in gdalwarp, NUM_THREADS=ALL_CPUS is used for multi-threaded compression only. So both options will not help. CACHE_MAX is the wrong notation. Use GDAL_CACHEMAX. This should definitely help. – pLumo Sep 25 '17 at 8:31
  • Thanks! I took a stab in the dark using the first two, and I don't really know where I got CACHE_MAX from. Using GDAL_CACHEMAX works perfectly – AlexVestin Sep 26 '17 at 6:04

I haven't used gdaldem commands, but my guess is that you won't be able to do much better without tiling and processing in parallel. I had a similar problem with other gdal commands taking forever, so I just tiled my region into individual tiffs and processed those in parallel. I'm not sure what "the same problem" in eurasia and islands regions means, so perhaps this isn't an option for you.

I left some code for tiling that may help you in another answer.

Then you can loop through each tile and use gdal.Create to write each one to a separate tiff (this could take a few hours and is parallelizable BUT you have to build a virtual raster on your original tiff if you want to run in parallel--see here). OR you could loop through the tiles and export the hillshade product rather than simply a smaller tif. Depends on what you want to do afterward.

A note: I am using the word "tiling" to refer to manually breaking a tiff into smaller tiffs. Although this is what gdal_tile does, it is not the same. It may be that you can initially manually tile to the size you eventually want and avoid gdal_tile entirely...

Once you've done that, you can process each tiff in parallel and re-stitch with either gdalmerge or just build a virtual raster (gdalbuildvrt) for easy display. The benefit of this approach is that if you want to do any further processing on your global dataset, you can do it all in parallel (assuming your processing doesn't require spatial context i.e. is not parallelizable).

  • This is how I went about doing the hillshading. Along with the help from @RoVo, running three processes and doing one file at a time finished in a couple of hours! – AlexVestin Sep 26 '17 at 6:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.