6

My Task
I need to convert 9 x 14Gb GeoTiff files to WGS84 geo projection. The rough size in pixels is about 200k x 100k.

The data-type is byte and it is not a subject to interpolation, it is some logical attribute. For example, if A = 20 and B = 30 does not mean that overall region is 25 - it has completely different meaning.

What I did

  • Started gdalwarp on Windows (takes lots of time, fails on some machines, never finishes on other)
  • Started gdalwarp on Ubuntu (Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz, 56Gb RAM, takes ~3 days for 50% and stops). Takes 100% of one core, low RAM and low HDD io, and also puts Potential thrashing on Band1 message in debug-log.
  • Modified arguments, e.g. CACHE_MAX, wa were set to some really big values.
  • The command I use now: gdalwarp -t_srs EPSG:4326 --config GDAL_CACHEMAX 1000 -wm 1000 --debug on $1 $2

Question
What are my other options?


I have successfully downloaded and compiled GDAL Beta 2. I have also placed entire source tif file in tmpfs (RAM). My current cmd is

gdalwarp -t_srs EPSG:4326 --config GDAL_CACHEMAX 99000 -wm 2000 -multi -co TILED=yes -wo NUM_THREADS=ALL_CPUS -wo STREAMABLE_OUTPUT=TRUE -oo GTIFF_DIRECT_IO=YES -oo GTIFF_VIRTUAL_MEM_IO=YES ram/2014/2014/2014_30m_cdls2.tif /mnt/test2.tif > test2.log 2>&1 &

It does ~40% pretty quick (5 mins). Other 5 minutes, that I waited, it did not provide any output. From that point, gdal stops using more than one core.

  • 1
    Have you tried the -multi option? Have you thought about subsetting your datasets and mosaicking them after conversion? Can you post the exact command you tried? – Kersten Jun 2 '15 at 7:20
  • @Kersten I have added the exact command. – Rustem Mustafin Jun 2 '15 at 7:27
  • 1
    For such huge files you should use -co BIGTIFF=YES option (trac.osgeo.org/gdal/ticket/2407), then maybe try Kersten's hint or take a look at GDAL 2.0 (beta) option of streaming operations- more info on page: gdal.org/frmt_gtiff.html – jzol Jun 2 '15 at 7:29
  • @Kersten The -multi option does little help -- it starts using 2 cores, but then the thrashing debug-log occurs, and only one of two gdalwarp process uses 100% one core -- as without that option. – Rustem Mustafin Jun 2 '15 at 7:37
  • 1
    What is your GDAL version? If it is lower than 2.0-dev you may suffer from the issue that is discussed in trac.osgeo.org/gdal/ticket/3120 and mail-archive.com/gdal-dev@lists.osgeo.org/msg04106.html. Your cachemax and wm settings are now 1000 megabytes which is very much. Your original settings with big numbers could have been in bytes and thus actually less, read trac.osgeo.org/gdal/wiki/ConfigOptions#GDAL_CACHEMAX – user30184 Jun 2 '15 at 11:59
4

I suggest tackling this using Virtual Raster Table (.vrt) format. How the end result is to be used will determine how many steps are needed.

Simplest possible case is the end product will be used by a GDAL or GDAL-aware program, create one .vrt in the desired projection and then use that in your final program:

gdalwarp -t_srs wgs84 -of vrt D:\gisdata\bigtiff*.tif test.vrt

This will probably only be feasible if the final program is using a small portion (view window) of the image at a time as the reprojection calculations happen at run time. This can be ameliorated by calculating overviews with gdaladdo:

gdaladdo -ro test.vrt 2 4 8 16 32 64

(nb: on my computer at moment this isn't creating external files like it used to. Might need to add some steps.)

Mapbox has an excellent and detailed article on a complex workflow using very large images with virtual rasters and multiple calculation steps: Super Sharp 50cm Pléiades Satellite Imagery on MapBox.com

2

It looks very strange, but warping with QGIS (which runs its embedded gdalwarp) is much faster!

I was able to process 14Gb file in 70 minutes, on windows, without much resource consumption. It still was not looking like it used multiple cores, but did the work, which is great. Also, it seems the same applies to gdal_translate.

Probably, they build gdal correctly, or something. The version is GDAL 1.11.2, released 2015/02/10.

Small note: on my files gdal_translate utility does all the work, but then places an error message: "incorrect file format: ". I ignored this message, because the file was correct.

  • this might be 64bit vs 32bit. In any case I'm glad you found a solution and thanks for sharing it. If you haven't already I would verify the end product has the right pixel size and wasn't resampled too coarsely (e.g. rule out the possibility Qgis was faster because it took a shortcut). – matt wilkie Jun 4 '15 at 17:27
  • 1
    @mattwilkie as a final solution, I did not start QGIS at all. I just started gdal* tools from its setup folder. – Rustem Mustafin Jun 5 '15 at 7:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.