# Seeking way to tile/pyramid 4 million images using gdal for geoserver WMS?

My scenario consist of 4 million geolocated images each measuring 1536x1536 pixels in size.

What is the most efficient way to pyramid these images?

My end goal is to serve these images using geoserver via WMS requests.

I have tried creating a virtual dataset of all images and putting them into gdal_retile.py. I have also tried merging each tile with a transparent "null" image that covers the globe then pyramid the result. Both methods seem like they will take forever and make me think there is a better way.

I also have the requirement of adding additional images in the future.

Is there an easy way to do this without having to regenerate everything again?

I am new to geoserver and gdal.

• Are they in one folder or grouped by folder? The most efficient is GDALAddO gdal.org/gdaladdo.html but you need some way to call this... you can't write it out 4 million times! – Michael Stimson May 21 '14 at 5:34
• They are separated into directories of 5000 images. I am assuming I can create a virtual dataset for 5000 images using gdalbuildvrt and pass the output into gdaladdo. It doesn't look like gdaladdo supports the --optfile option. – PCL May 21 '14 at 5:44
• No, but the VRT contains all the links that GDALAddO needs. This creates a single overview for the 5k images. – Michael Stimson May 21 '14 at 22:24
• What are your requirements in terms of display? Do you need to see the whole mosaic in a single WMS request? Or are you going to limit its visibility based on the request scale denominator? – Andrea Aime Jul 12 at 8:17

If you are in a windows environment you can create a batch file:

@echo off
for /f %%A in ('dir "c:\your_Path\*.tif" /b/s') do (
echo %%A
"c:\path\to\GDAL\bin\gdaladddo" -ro %%A 2 4 8 16 32
)


This is a batch file that says "for this folder and every subfolder, every file with the '.tif' extension run gdaladdo". You will need to change c:\path\to\GDAL\bin to match your GDAL install location, you will also need to change .tif to the actual extension of the files.

Change the c:\your_path\ to where your folder of 5k is and run the batch file. You can run one per CPU thread in a separate cmd window. There are other options that can be added but let's just keep it simple.

important if any of the paths contain spaces you will need to quote them: "c:\path with spaces\files\" or DOS will spit the dummy, if the file names themselves contain spaces you will need to do something in python.

• So what you're saying is that I should use gdaladdo rather than gdal_retile? I would need to add internal overlays for all 4 million images then use gdaltindex to create a shape file that geoserver can read via the imagemosaic plugin. How will geoserver behave with so many files? Also, how can I add images without having to rebuild the entire shape file every time? – PCL May 21 '14 at 6:43
• Yes. gdal_retile is one process using one core, as stated the process can be tasked with 1 cmd per core with the bottleneck being HDD access. For internal overviews take out the -ro switch, this one says "don't modify the image but create an external overview". Gdaltindex should be reasonably quick as it's just describing, sorting, indexing and writing a little bit of data to the drive. I don't know about imagemosaic plugin but VRT may ignore internal overviews like it does with ECW tiles and want its own overview. – Michael Stimson May 21 '14 at 22:17
• Anything involving 4 million images is going to be slow and tedious. Is there any change of mosaicing the folder of 5k images? GdalBuildVRT followed by GDAL_Translate would usually do. – Michael Stimson May 21 '14 at 22:20
• I performed these operations on one set of 5k images and the resulting file is very large (100's of GB without compression applied in gdal_translate, 10's of GB with compression). Can geoserver cope with files of this size, or do I need to pyramid or add internal tiling and overviews? Also, how to I go about combining each set of 5k images to add them as a single layer in geoserver? I could possibly create a shape file via gdaltindex and add an ImageMosaic layer. – PCL May 22 '14 at 0:58
• Always a good idea to pyramid an image that is to be used more than once. I can't tell you about geoserver implicitly but if the data is tiled in moderate pieces and pyramids are built (so long as they are compatible with the system) every system that I've used would have no problem. Hypertiles, VRT and I assume ImageMosaic layer are good at handling huge image sets as they only need to access a few tiles at any given time (zoomed in) and a pyramid when zoomed out. – Michael Stimson May 22 '14 at 1:05