Converting huge geotiffs to tilepyramid

Question

I'm trying to convert a 20 GB LZW compressed geotiff to a tilepyramid using gdal2tiles. Unfortunately, gdal2tiles seems not to be able to handle this, after running it for 10 hours it still hasn't reached the first "dot" yet. I tried this with a smaller (1GB) file and it worked fine. My question now is, is there any other software that allows me to do this (commercial or open source)? One can create tiles from GlobalMapper, but I cannot load the 20GB file into memory.

Answer 1

I had the same problem just a few weeks ago. For once, you use the option

--config GDAL_CACHEMAX 2047

(must stay below 2048 for the time being!) And second, you can copy the gdal2tiles.py script to another location and alter the for-loop which iterates over the tiles to count backward. in my version of gdal2tiles that’s at line 1174:

for ty in range(tmaxy, tminy-1, -1):

change to:

for ty in range(tminy, tmaxy+1):

then use both scripts with the same arguments at the same time and add -e (i.e. continue, don’t delete already existing files) Note, that you have to quit one of the scripts as they come towards 50% completion (of the base tiles), or you’ll get broken tiles. If you forget, just search for all files with size 0, delete them and start all over - with the -e switch ;)

hope that helped

Answer 2

If you are using a computer with multicore processor you can find Parallel GDAL2Tiles (ppgdal2tiles) useful. In this kind of computers ppgdal2tiles should run faster than the "single-core" version. Quoting ppgdal2tiles web page:

"The standard version of GDAL2Tiles is using only one processing core at a time.

In case you have a computer with more processors (or one of the Mutli-Core processors from Intel or AMD) then several processes can be started to use the full computing power of your computer to finish the tile rendering as fast as possible."

Answer 3

I would hope that FME would be able to do that. It certainly can do the tiling - and I'm confident it can handle the large file sizes. The raster pyramiding part is pretty straightforward, see Raster Pyramiding example in the samples and demos pages.

You can get an evaluation copy to try it on first

Mark Ireland, Product Evangelist, Safe Software

Answer 4

Try the second patch from this issue for a parallel version of gdal2tiles.py using the 'multiprocessing' module: http://trac.osgeo.org/gdal/ticket/4379

I got a 5.5 x speedup on a 6 core CPU.