0

I am creating Cloud Optimized GeoTIFFs using gdal from a folder of TIFF files in GDAL for an entire county. After creating a mosaic of these files (in EPSG:6551) using gdalbuildvrt I can create a COG from the mosaic in a few days, maybe 3-5. The output is amazing - small file size and crisp imagery. However, when I try to use gdalwarp to re-project the imagery, either the mosaic or the COG, to WebMercator the process starts, writes a small amount of data, but then appears to hang, though gdalwarp is still consuming memory and CPU.

I am wondering if there is a way to either a) debug gdal to make sure it is not hanging and is indeed running, or b) to speed up the warp process. I am already using all the maxcache, multi, NUM_THREADS settings I can find. I also tried to warp each TIFF individually, which works but creates transparent artifacts in the end product on the edges of the individual TIFF files.

It may be that this is the best I can expect from an 8th Gen i5 with 6 Cores and 32GB ram...

Warp settings

-et 0 -overwrite -wo NUM_THREADS=ALL_CPUS -multi -r lanczos ^

COG Settings

-of COG -co RESAMPLING=CUBIC -co COMPRESS=JPEG -co BIGTIFF=YES -co NUM_THREADS=ALL_CPUS -co QUALITY=85 ^
--config GDAL_CACHEMAX 9999 ^
--config GDAL_NUM_THREADS ALL_CPUS
8
  • 1
    While not an answer to this particular problem, I am going to try and use the -tap setting along with various -tr settings along side warping each tiff file. These ideas came from here - gis.stackexchange.com/questions/107903/….
    – Malcolm
    Sep 22, 2021 at 18:27
  • Try without --config GDAL_CACHEMAX 9999 , usually it just makes things slower by forcing GDAL to keep bigger cache than it really needs. Using -tap for individually warped images is a good idea.
    – user30184
    Sep 22, 2021 at 18:33
  • -tr did not solve the transparent artifacts between tiffs.
    – Malcolm
    Sep 22, 2021 at 19:49
  • I wonder if it has to do with JPEG compression. I understand that it is not an option for your whole task but you could make a test with handful of tiffs by using lossless compression "deflate". And -tr was not what is important but -tap, target aligned pixels. Option is normally used with -tr, but I suppose that's what you did.
    – user30184
    Sep 22, 2021 at 20:18
  • One more option could be to warp the VRT or COG into biggish rectangular tiles by using -projwin. Find out how large area can be converted in a reasonable time. There may be logarithmic grow in time/area ratio.
    – user30184
    Sep 23, 2021 at 5:59

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.