You need to tile the image and add overviews so that the whole image is never read into memory at the same time. GeoServer provides an image pyramid datastore for this purpose.
I wrote these notes describing how I set this up on my machine. The key step is to use GDAL to build the pyramid using the following command:
gdal_retile.py -v -r bilinear -levels 8 -ps 2048 2048 -co \
"TILED=YES" -targetDir bmpyramid \
- -v: verbose output, allows the user to see each file creation scroll by, thus knowing progress is being made (a big pyramid construction
can take hours)
- -r bilinear: use bilinear interpolation when building the lower resolution levels. This is key to get good image quality without
asking GeoServer to perform expensive interpolations in memory
- -levels 8: the number of levels in the pyramid
- -ps 2048 2048: each tile in the pyramid will be a 2048x2048 GeoTIFF
- -co “TILED=YES”: each GeoTIFF tile in the pyramid will be inner tiled
- (Not used here) -co “COMPRESS=JPEG”: each GeoTIFF tile in the pyramid will be JPEG compressed (trades small size for higher performance, try out it
without this parameter too)
- -targetDir bmpyramid: build the pyramid in the bmpyramid directory. The target directory must exist and be empty
- bmreduced.tiff: the source file
Utility notes for gdal_retile.py.