Hosting 100 GB of MrSID files in GeoServer

Question

I've been given 100 GB worth of MrSID files (about 2,300) and been asked to create one map with them that can be hosted in GeoServer.

I originally attempted to put all of these tiles into a layer in ArcMap but after adding about 1/3 of them, the program just crashed. After that, I did a lot of research about MrSID files and GeoTIFF files and based on the limitations of my own system (500GB HD) and the target environment, hit a bit of a dead end.

Thus the move to GeoServer. The goal is to be able to host the data, in what format I'm still not sure, and only access the portion we need in the client application.

I was starting to think that a Mosaic Dataset through ArcMap was the best approach, but I'm not sure if that can be integrated into GeoServer.

How would you go about this problem?

Answer 1

More than a 1000 (or so) files will be too many for an image mosaic in GeoServer so you will want to create an Image Pyramid. The generally recommended approach is to convert from "exotic" formats (such as MrSid) to a compressed, tiled GeoTiff.

So I would use gdal_translate to convert each MrSid file into a GeoTiff

gdal_translate \
  -co COMPRESS=JPEG \
  -co PHOTOMETRIC=YCBCR \
  -co TILED=YES \
  in.sid out.tif

Then I would create a virtual raster using gdalbuildvrt

gdalbuildvrt mosaic.vrt *.tif

and then use gdal_retile.py to create a pyramid

mkdir pyramid

gdal_retile.py -v -r bilinear -levels 4 -ps 2048 2048 \
 -co "TILED=YES" -co "COMPRESS=JPEG" -co PHOTOMETRIC=YCBCR \
 -targetDir pyramid mosaic.vrt

GeoServer should then import the pyramid with no issues. Once loaded you can server WMS images of the imagery or if you need to do actual calculations of the data then the WCS can be used.

Answer 2

Per the GeoServer documentation, GeoServer has an ImageMosaic data store which will support these MrSid files, providing the GDAL extension is also installed.

Given that you already mentioned GeoServer in your problem statement, I believe there is a suitable solution for you and your Sid files using this technology.

Alternately, LizardTech's ImageServer would also serve a mosaic of MrSids.

Either way, the resulting raster service should behave as you describe -- only "accessing" the parts of interest from a client environment.

A key point here is what kind of "access" is expected...if the client environment only wants to "view" portions of the image mosaic then you are on the right track. However, if the client environment needs to perform raster analytics, then the choice of client becomes every bit as important as the choice of image service technology.

Answer 3

Using Ian Turton's answer I was able to write a script that converted the MrSid files to GeoTiffs and then create a pyramid. I couldn't get GDAL to work with the Lizard Tech SDK, so I had to use GeoExpress Command Line Tools to convert the file.

I also could not include the -co PHOTOMETRIC=YCBCR command because it caused an error - 0error 6: photometric=ycbcr requires a source raster with only 3 bands (rgb). I'm assuming this was a result of using a different conversion process.

This is my batch script:

    SET newTiffsTargetDir=%USERPROFILE%\Desktop\newTiffs
    mkdir %newTiffsTargetDir%

    for %%i in (*.sid) do mrsidgeodecode -wf -i %%i -o %newTiffsTargetDir%\%%~ni.tif

    cd %newTiffsTargetDir%

    SET compressed_tiled_TargetDir=%USERPROFILE%\Desktop\compressed_and_tiled
    mkdir %compressed_tiled_TargetDir%

    for %%i in (*.tif) do gdal_translate\
    -co COMPRESS=JPEG -co TILED=YES\ 
    %%i %compressed_tiled_TargetDir%\%%~ni_tiled.tif & del %%I

    cd %compressed_tiled_TargetDir%

    gdalbuildvrt mosaic.vrt *.tif 

    SET pyramidDir=%USERPROFILE%\Desktop\pyramid
    mkdir %pyramidDir%

    gdal_retile.py -v -r bilinear -levels 4 -ps 2048 2048\ 
    -co "TILED=YES" -co "COMPRESS=JPEG"\
    -targetDir %pyramidDir% mosaic.vrt

I deleted the first GeoTiff that was created just as a sort of cleanup operation.

I've yet to try with all 2,300 files, but this works for a smaller subset and uploads into GeoServer.

Answer 4

For publishing Orthophotos or Aerials or Satellite Imagery I like to make XYZ or TMS PNG or JPG Tiles. Server those out of AWS S3 Bucket. Skip GeoServer, GeoWebCache and PostGIS and deliver an Google XYZ Tile URL or WMTS URL right from S3 or go with a tile server. You can create a VRT that combines the many SID's and then make raster tiles. Do you have access to QGIS Desktop or Global Mapper? if not just use GDAL_TRANSLATE -of MBTILES sid.vrt output.mbtiles GDALADDO to create zoom levels/pyramids if you want a folder of tiles https://github.com/pbarry/MBTiles-extractor use this python script to extract your mbtiles.