I am running GeoServer 2.8 on Ubuntu 14 as a WAR in Tomcat 7. I have an 8 core machine with 16GB of RAM and have allowed a generous Heap size of 4GB. I have JAI installed and enabled. Meta tiling is set to 8x4 and I have allowed 3GB for rendering so memory should not be a problem.
I have a layer-group with scale-dependent styles. The various layers transition from simple global data to OSM. The problem is that at the global or low-zoom levels, Geoserver seems to be attempting to load and process all data in all layers even though most will not be rendered. When you consider OSM - that is a LOT of data! I have tried to ease the pain by making views of low-zoom data and high-zoom data (e.g. for roads the low-zoom misses out a lot of the detail and minor roads). This does not help and possibly compounds the issue. Looking at memory consumption as a whole, HTOP shows that Java is barely using any of the 4GB heap allowed. So throwing memory at GeoServer doesn't help.
The irony is that rendering is fine at mid to high zoom levels, even though now the OSM data is visible, presumably as the data requested is much more limited. All my OSM layers are spatially indexed and clustered - so no problem here. Even so, that should be irrelevant at low zoom levels as the OSM layers are not visible in my styles.
At the moment I am working around the problem by setting my render times for WMS to 1 hr and seeding the cache (very slowly - until I get to intermediate zoom levels when the data load reduces to sane amounts). This is not a great solution, especially if I make a minor change to my layer-group.
I want a single layer-group to keep things simple at the client end. The best solution, as I see it, is to implement a scale-dependent 'switch' in the layer-group definition but that requires GeoServer-Developer input. However, what practical methods do people suggest to improve performance in the meantime (I have already read
GeoServer on Steroids and implemented everything relevant in there).
The problem was due to a corrupted style that cause a large volume of very small polygons to be drawn at all levels - which explains the observed results. fixing the style, fixes the problem. However, this has highlighted what appears to be a serious memory leak following failures, which progressively causes Geoserver to become less responsive and only a restart of Tomcat combined with a nasty
killall -9 Java call fixes (just restarting Tomcat leaves the leaks as checking for leaks in Tomcat manager will confirm).
In my logs I periodically have the following error:
ERROR [org.geotools.map] - Call MapContent dispose() to prevent memory leaks
The error is never reported singly , but always multiple times (into double figures sometimes) and is associated with a few DEBUG logs especially:
DEBUG [org.vfny.geoserver.responses.wms.map] - setting to transparent 2015-08-08 08:59:48,448 WARN [org.geotools.renderer.lite.gridcoverage2d] - Could not reduce the grid geometry inside the valid area bounds: ReferencedEnvelope[-1.7976931348623157E308 : 1.7976931348623157E308, -85.0 : 85.0] Grid geometry isGridGeometry2D[GridEnvelope2D[6365..7293, 7345..7818],
DEBUG [org.geoserver.security.IncludeQueryStringAntPathRequestMatcher] - Checking match of request : 'Path: /web/, QueryString: x=0bkVEjecmbChX7z8UByFaQCm3CLIUMoGOhTdEK9-vGhb41MvY3jdDsgkHKnbSgYXWRKtVX*xG5ASH1XP*FxePw'; against '/web/**' 2015-08-09 16:39:43,123 DEBUG [org.geoserver.security.IncludeQueryStringAntPathRequestMatcher] - Matched Path: /web/, QueryString: x=0bkVEjecmbChX7z8UByFaQCm3CLIUMoGOhTdEK9-vGhb41MvY3jdDsgkHKnbSgYXWRKtVX*xG5ASH1XP*FxePw with /web/** 2015-08-09 16:39:43,123 TRACE [org.geoserver.ows.OWSHandlerMapping] - No handler mapping found for [/web/] 2015-08-09 16:39:43,123 TRACE [org.geoserver.ows.OWSHandlerMapping] - No handler mapping found for [/web/] 2015-08-09 16:39:43,123 TRACE [org.geoserver.ows.OWSHandlerMapping] - No handler mapping found for [/web/] 2015-08-09 16:39:43,123 TRACE [org.geoserver.ows.OWSHandlerMapping] - No handler mapping found for [/web/] 2015-08-09 16:39:43,126 DEBUG [org.geoserver] - Thread 1587 locking in mode WRITE 2015-08-09 16:39:43,126 DEBUG [org.geoserver] - Thread 1587 got the lock in mode WRITE 2015-08-09 16:39:45,927 ERROR [org.geotools.map] - Call MapContent dispose() to prevent memory leaks
This question can be closed as it has now strayed from GIS into programming and this isn't the forum for it - I'll mark the only answer as the solution.