Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a 25 large geoTiff files, representing a DEM, to be published via Geoserver in order to have OGC WCS. The size of each uncompressed file ranges between 235 MB and 6GB. What would it be the best method to follow considering performance? I read a lot of documentation about image mosaic plugin and imagePyramid plugin, but I do not still figure out what to do, whether adding inner tiles and overviews on all geotiffs then publish using image mosaic or merging those file into one file then using pyramid plugin.

share|improve this question
    
what are your expected request patterns? lots of requests for the whole area or many requests for small areas? – iant Feb 10 at 16:00
    
I am still not sure about this, but I can imagine it would fit the first option: lots of requests for the whole area. – defelix Feb 10 at 16:05
up vote 2 down vote accepted

Based on the advice in "GeoServer on Steroids" I would aim for a mosaic of GeoTiffs. Page 8 clearly states to choose a mosaic when:

  • A single file gets too big (inefficient seeks, too much metadata to read, etc..)
  • Single granules can be large
  • Use Tiling + Overviews + Compression on granules

but suggests moving to pyramids when

  • Tremendously large dataset
    • Too many files / too large files
  • Need to serve at all scales
    • Especially low resolution

So I would use GDAL to split them into a collection of GeoTiffs with tiles and overviews set and then drop them all into an Image Mosaic. Some experimentation could be useful to see what the best size of GeoTiff v. Number of granules is, but I'd err towards keeping it below a 100 if you are doing a lot of full views.

Update

To split up a GeoTiff you can use a script like:

#!/bin/bash
# If I was really bored I'd work out how to parse the gdalinfo output
#Upper Left  (     -25.000, 1240025.000) (  9d22'14.87"W, 60d50'27.75"N)
#Lower Left  ( -25.0000000, -25.0000000) (  7d33'24.37"W, 49d45'57.40"N)
#Upper Right (  660025.000, 1240025.000) (  2d48'16.84"E, 60d57'27.18"N)
#Lower Right (  660025.000,     -25.000) (  1d37' 1.96"E, 49d50'34.37"N)
#Center      (  330000.000,  620000.000) (  3d 6'26.52"W, 55d28' 7.68"N)

x=0
y=0
mx=700000
my=1300000
step=10000

for ((i=0; i < $mx; i+=$step)); do
  for ((j=0; j < $my; j+=$step)); do
    nx=$(($i + $step))
    ny=$(($j + $step))
    echo "gdal_translate -projwin" $nx $ny $i $j
  done
done
share|improve this answer
    
Thanks a lot @iant. just to be more precise and avoid misunderstanding you are suggesting to run gdal_translate and gdaladdo for each geotiff in order to add inner tiling and overviews, then to publish against image Mosaic, isn' t it?Should I compress the geotiffs? – defelix Feb 10 at 16:26
1  
yes, once you have split them up into smaller tiffs using gdal_translate with -projwin ulx uly lrx lry - hard to be precise without seeing the input datasets – iant Feb 10 at 16:31
    
always compress, tile and add compressed overviews – iant Feb 10 at 16:31
    
So I need to use for each file gdal_translate with -projwin ulx uly lrx lry In order to split them, but is there a way for splitting all files in one shot using something like a bacth command? – defelix Feb 10 at 16:51
    
added a script - this is veering into consultancy now, email me for prices :-) – iant Feb 10 at 17:08

Ciao, an observation on number VS size of files.

Opening more than 10 to 20 file to respond to a request would hinder scalability of WMS and WCS. On the other side having Geotiff that are too large (hard to say but >> 20gb as a start) would make the internal structure too big for fast serving.

As such you need to balance size and # of files to read on each request. By experience in your case you could simple reprocess each file and then mosaic. If you want you might want to first merge to reduce the number of files to use. Use compression to reduce the size if possible.

share|improve this answer
    
I compressed each file using LZW, now the size ranges between 20MB to 3.5GB, having 25 files.Maybe I could try adding tiles and overviews for each geotifft then use image mosaic directly without further splitting them? – defelix Feb 10 at 17:07
    
that's the way to go I guess. you can also try do a first step to reduce the # of files by increasing the size with a merge. – simogeo Feb 10 at 17:43
    
Hi,I did first trial with inner tiles and overviews, splitting biggest geotiff files into smaller ones, but now I have 43 files and it seems Geoserver is not able to render, I obtain a white map, but if I zoom in a bit I can see just some tiles rendered.should I try to merge the geotiffs? – defelix Feb 16 at 16:45
    
Ehm, did you read my comment above? "Opening more than 10 to 20 file to respond to a request would hinder scalability" so you decided to split into 43 files? – simogeo Feb 16 at 19:42
    
@simongeo I am considering your comment, but the problem is that I have files whose size is really big, more than 6 Gb, so it is difficult to find the right compromise.I already tried without splitting biggest files into smaller tiles, but without great success.As you have suggested the key is to find good balance between # geotiffs and size.Now I will try to merge the smallest geotiffs in order to get less files. – defelix Feb 17 at 15:29

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.