Render imageMosaic of palleted GeoTIFFs via WMS from GeoServer

Question

I'm ultimately attempting to serve an external ImageMosaic of GeoTIFFs via WMS. The individual tiffs are pca hillshades, generated using r.shaded.pca in GRASS 7 GIS, exported via GDAL to UInt16 and are paletted.

Example output from gdalinfo:

Driver: GTiff/GeoTIFF
Files: /srv/mal_data/external_data/processed_lidar/spca/oxfordshire_merged-products-dtm_1m_spca_20200611_215851.tif
       /srv/mal_data/external_data/processed_lidar/spca/oxfordshire_merged-products-dtm_1m_spca_20200611_215851.tif.aux.xml
Size is 39450, 48246
Coordinate System is:
PROJCS["OSGB 1936 / British National Grid",
    GEOGCS["OSGB 1936",
        DATUM["OSGB_1936",
            SPHEROID["Airy 1830",6377563.396,299.3249646,
                AUTHORITY["EPSG","7001"]],
            TOWGS84[446.448,-125.157,542.06,0.15,0.247,0.842,-20.489],
            AUTHORITY["EPSG","6277"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4277"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",49],
    PARAMETER["central_meridian",-2],
    PARAMETER["scale_factor",0.9996012717],
    PARAMETER["false_easting",400000],
    PARAMETER["false_northing",-100000],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["Easting",EAST],
    AXIS["Northing",NORTH],
    AUTHORITY["EPSG","27700"]]
Origin = (434733.000000000000000,230808.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Metadata:
  AREA_OR_POINT=Area
  TIFFTAG_SOFTWARE=GRASS GIS 7.6.1 with GDAL 2.4.2
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (  434733.000,  230808.000) (  1d29'39.48"W, 51d58'27.21"N)
Lower Left  (  434733.000,  182562.000) (  1d29'56.85"W, 51d32'25.47"N)
Upper Right (  474183.000,  230808.000) (  0d55'12.02"W, 51d58'13.29"N)
Lower Right (  474183.000,  182562.000) (  0d55'49.09"W, 51d32'11.77"N)
Center      (  454458.000,  206685.000) (  1d12'39.39"W, 51d45'20.70"N)
Band 1 Block=512x512 Type=UInt16, ColorInterp=Palette
  Description = tempras_pca
  NoData Value=65535
  Overviews: 19725x24123, 9863x12062, 4932x6031, 2466x3016, 1233x1508, 617x754
  Metadata:
    COLOR_TABLE_RULES_COUNT=1024
    COLOR_TABLE_RULE_RGB_0=0.000000e+00 3.100000e+01 0 0 0 255 0 0
    COLOR_TABLE_RULE_RGB_1=3.200000e+01 6.300000e+01 0 8 0 255 8 0
    COLOR_TABLE_RULE_RGB_10=3.200000e+02 3.510000e+02 0 82 0 255 82 0
    COLOR_TABLE_RULE_RGB_100=3.200000e+03 3.231000e+03 0 32 24 255 32 24
    COLOR_TABLE_RULE_RGB_1000=3.200000e+04 3.203100e+04 0 65 255 255 65 255
    COLOR_TABLE_RULE_RGB_1001=3.203200e+04 3.206300e+04 0 74 255 255 74 255
    COLOR_TABLE_RULE_RGB_1002=3.206400e+04 3.209500e+04 0 82 255 255 82 255
    COLOR_TABLE_RULE_RGB_1003=3.209600e+04 3.212700e+04 0 90 255 255 90 255
    COLOR_TABLE_RULE_RGB_1004=3.212800e+04 3.215900e+04 0 98 255 255 98 255
    COLOR_TABLE_RULE_RGB_1005=3.216000e+04 3.219100e+04 0 106 255 255 106 255
    COLOR_TABLE_RULE_RGB_1006=3.219200e+04 3.222300e+04 0 115 255 255 115 255
    COLOR_TABLE_RULE_RGB_1007=3.222400e+04 3.225500e+04 0 123 255 255 123 255
    COLOR_TABLE_RULE_RGB_1008=3.225600e+04 3.228700e+04 0 131 255 255 131 255
    COLOR_TABLE_RULE_RGB_1009=3.228800e+04 3.231900e+04 0 139 255 255 139 255
    COLOR_TABLE_RULE_RGB_101=3.232000e+03 3.263000e+03 0 41 24 255 41 24
    COLOR_TABLE_RULE_RGB_1010=3.232000e+04 3.235100e+04 0 148 255 255 148 255
    COLOR_TABLE_RULE_RGB_1011=3.235200e+04 3.238300e+04 0 156 255 255 156 255
    COLOR_TABLE_RULE_RGB_1012=3.238400e+04 3.241500e+04 0 164 255 255 164 255
    COLOR_TABLE_RULE_RGB_1013=3.241600e+04 3.244700e+04 0 172 255 255 172 255
    COLOR_TABLE_RULE_RGB_1014=3.244800e+04 3.247900e+04 0 180 255 255 180 255
    COLOR_TABLE_RULE_RGB_1015=3.248000e+04 3.251100e+04 0 189 255 255 189 255
    COLOR_TABLE_RULE_RGB_1016=3.251200e+04 3.254300e+04 0 197 255 255 197 255
    COLOR_TABLE_RULE_RGB_1017=3.254400e+04 3.257500e+04 0 205 255 255 205 255
    COLOR_TABLE_RULE_RGB_1018=3.257600e+04 3.260700e+04 0 213 255 255 213 255
    COLOR_TABLE_RULE_RGB_1019=3.260800e+04 3.263900e+04 0 222 255 255 222 255
...

If I add a single TIFF as a data store the layer appears to work correctly, with the correct Coverage Band Details automatically populating the configuration:

The layer previews nicely via OpenLayers within the GeoServer GUI, though I note it automatically selects JPEG as the format — if I select a different image format then nothing is displayed.

I can get this layer to display correctly in QGIS if I explicitly set the image encoding to JPG using the Data Source Manager, though the no data values are displayed as opaque white.

My imageMosaic of multiple files displays a solid white block however (though I can get the mosaic to display correctly if it only contains a single TIFF). I think there is an issue with transparency and that no data values from multiple granules are displaying solid white and obscuring the visible sections of data.

Answer 1

Solution was to first us pct2rgb.py to convert images to multiband rasters. Then use gdal_translate to add overviews and compression. Images now overlay correctly in WMS and can be rendered as PNG with transparency using QGIS. Hacky Python script below:

#!/usr/bin/env python
import sys
import os
from osgeo import gdal
import datetime

def my_print(text):
    sys.stdout.write('\n' + str(text))
    sys.stdout.flush()
    
def addOverviews(file):
    currentDT = datetime.datetime.now()
    my_print(str(currentDT))
    my_print('Adding overviews...')
    InputImage = file
    Image = gdal.Open(InputImage, 1) # 0 = read-only, 1 = read-write.
    gdal.SetConfigOption('COMPRESS_OVERVIEW', 'DEFLATE')
    Image.BuildOverviews("AVERAGE", [2,4,8,16,32,64])
    del Image
    currentDT = datetime.datetime.now()
    my_print(str(currentDT))
    my_print('Overviews added.')

# expand RGBA
outputName = 'my_tiff.tif'
cmd = 'pct2rgb.py -of GTiff -rgba ' + \
outputName + ' ' + outputName[:-4] + '_rgba.tif'
print(cmd)
os.system(cmd)
cmd = 'gdal_translate '+ outputName[:-4] + '_rgba.tif ' + \
outputName + ' -co compress=LZW -co TILED=YES '+\
'-co BLOCKXSIZE=512 -co BLOCKYSIZE=512'
print(cmd)
os.system(cmd)
cmd = 'rm '+ outputName[:-4] + '_rgba.tif '
os.system(cmd)
addOverviews(outputName)