Slow initial loading of layers in Geoserver from huge COGs stored in a AWS S3

Question

I am working with huge COGs (around 500GB) stored on my private S3 bucket. I am trying to publish these COGs as GeoTiff layers using COG extension in Geoserver. The Geoserver is deployed on an EC2 instance with 8 vCPUs and 16Gb of RAM. The projection system of COGs is WGS84 (EPSG:4326) and I want to access the WMS in WGS84.

The layers are published successfully, but when I want to access the WMS of these layers, the initial loading of the layers take about a couple of minutes. But when they are loaded, everything works smoothly and tiles are loaded pretty quick. Also, when the COG is smaller in size (less than 2GB), the initial loading is pretty good.

I realized that first thing Geoserver does, is to load the header of the COG and then access the inner tiles. By default, Geoserver reads first 16KB of COGs to access the header. I have tried to increase it.geosolutions.cog.default.header.length so that a larger portion of the COG is read initially. But, it ended up slower loading of COGs.

The Geoserver version is: 2.22.3
Web server is: Tomcat 9
OS is: Ubuntu

Did anybody have experience working with huge COGs? Any solutions?

Answer 1

This parameter:

it.geosolutions.cog.default.header.length

is in bytes. I had set it in Kbytes (256). As a result, I had mistakenly decreased the length, so the Geoserver had to do more iterations to read the full header. Now that I have set it to 262144 (bytes), the initial load is completed in couple of seconds.