I need to convert 9 x 14Gb GeoTiff files to WGS84 geo projection. The rough size in pixels is about 200k x 100k.
The data-type is byte and it is not a subject to interpolation, it is some logical attribute. For example, if
A = 20 and
B = 30 does not mean that overall region is 25 - it has completely different meaning.
What I did
- Started gdalwarp on Windows (takes lots of time, fails on some machines, never finishes on other)
- Started gdalwarp on Ubuntu (Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz, 56Gb RAM, takes ~3 days for 50% and stops). Takes 100% of one core, low RAM and low HDD io, and also puts
Potential thrashing on Band1message in debug-log.
- Modified arguments, e.g. CACHE_MAX, wa were set to some really big values.
- The command I use now:
gdalwarp -t_srs EPSG:4326 --config GDAL_CACHEMAX 1000 -wm 1000 --debug on $1 $2
What are my other options?
I have successfully downloaded and compiled GDAL Beta 2. I have also placed entire source tif file in tmpfs (RAM). My current cmd is
gdalwarp -t_srs EPSG:4326 --config GDAL_CACHEMAX 99000 -wm 2000 -multi -co TILED=yes -wo NUM_THREADS=ALL_CPUS -wo STREAMABLE_OUTPUT=TRUE -oo GTIFF_DIRECT_IO=YES -oo GTIFF_VIRTUAL_MEM_IO=YES ram/2014/2014/2014_30m_cdls2.tif /mnt/test2.tif > test2.log 2>&1 &
It does ~40% pretty quick (5 mins). Other 5 minutes, that I waited, it did not provide any output. From that point, gdal stops using more than one core.