0

I set up GeoServer (2.15) to run from a docker instance and mounted the data_dir externally. I then configured my server with a workspace, layers, styles and layer groups.

The docker instance was stopped and started as I resized my environment to have more memory. However, now GeoServer has lost some of the config when restarted.

I can see the workspace and styles have survived but the layers and layergroups have not. Clicking layergroups in the backend shows a page of Java errors.

Can I recover this situation by starting afresh and just copying in my workspace? I tried using the styles/workspaces/layergroups directories but as I said got the Java errors.

FINAL EDIT

OK a significant edit to the question as it appears the problem is linked to an environment change.

I can reproduce as follows:-

  • create Amazon AWS EC2 instance
  • setup clean docker instance and install geoserver/geocache within instance, mount data_dir externally
  • configure geoserver via GUI
  • shutdown docker instance [sudo docker stop --time=60 $(sudo docker ps -q);]
  • resize EC2 instance (as a result the IP address may change)
  • restart docker instance [sudo docker start ]

Now Geoserver will not start but Tomcat will. I note the following errors in catalina.2019-05-09.log

INFO [java-sdk-http-connection-reaper] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [org.apache.http.pool.AbstractConnPool$3]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
 java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.http.pool.AbstractConnPool$3]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1372)
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1360)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1219)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1180)
    at org.apache.http.pool.AbstractConnPool.closeIdle(AbstractConnPool.java:622)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.closeIdleConnections(PoolingHttpClientConnectionManager.java:423)
    at com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:172)

So I restarted Tomcat and now Geoserver "almost runs" but dies with these errors:-

org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [geoserver] appears to have started a thread named [FileSystemWatcher-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 java.lang.Thread.run(Thread.java:748)
09-May-2019 09:05:11.896 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [geoserver] appears to have started a thread named [java-sdk-http-connection-reaper] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Thread.sleep(Native Method)
 com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:157)
09-May-2019 09:05:11.897 SEVERE [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [geoserver] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@18757ee0]) and a value of type [org.springframework.security.core.context.SecurityContextImpl] (value [org.springframework.security.core.context.SecurityContextImpl@ffffffff: Null authentication]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
09-May-2019 09:05:11.898 SEVERE [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [geoserver] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@5d2b5543]) and a value of type [org.geowebcache.storage.CompositeBlobStore.StoreSuitabilityCheck] (value [EXISTING]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
09-May-2019 09:05:11.903 INFO [localhost-startStop-1] java.util.prefs.FileSystemPreferences$1.run Created user preferences directory.
09-May-2019 09:05:11.915 SEVERE [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [geoserver] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@6e1ab278]) and a value of type [org.geotools.referencing.operation.projection.TransverseMercator.Provider] (value [PROJECTION["Transverse_Mercator"]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
09-May-2019 09:05:11.916 SEVERE [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [geoserver] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3f670f47]) and a value of type [org.geotools.referencing.wkt.Formatter] (value []) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
09-May-2019 09:05:11.928 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /usr/local/tomcat/webapps/geoserver has finished in 73,038 ms
09-May-2019 09:05:11.933 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-apr-8080"]
09-May-2019 09:05:11.947 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-apr-8009"]
09-May-2019 09:05:11.951 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 74219 ms
09-May-2019 09:05:30.953 INFO [java-sdk-http-connection-reaper] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [org.apache.http.pool.AbstractConnPool$3]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
 java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.http.pool.AbstractConnPool$3]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1372)
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1360)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1219)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1180)
    at org.apache.http.pool.AbstractConnPool.closeIdle(AbstractConnPool.java:622)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.closeIdleConnections(PoolingHttpClientConnectionManager.java:423)
    at com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:172)
  • just restore the data directory from your backup and be more careful stopping docker in future. – Ian Turton May 8 at 8:31
  • All very well in hindsight but we were experimenting rather than creating a production instance. Copying the data from the backup dir does not work as I stated above. The bit that I'm puzzled by is that creating a new geoserver instance and copying in workspaces dir only restores the workspace and styles. Why are the layers and stores not copied in? – CitizenFish May 8 at 8:55
  • You need to copy all of the data dir in not just bits and peices – Ian Turton May 8 at 9:11
  • Unfortunately that is exactly what breaks it all. – CitizenFish May 8 at 10:15
  • then your back up is corrupt, try with a clean install – Ian Turton May 8 at 12:18
0

OK after a lot of frustrated digging around this turned out to be an issue with the docker build not the geoserver configuration.

Leaving this here just in case anyone uses the image https://github.com/kartoza/docker-geoserver

The image uses a startup script to rebuild which requires internet access, fine in 99.99% of cases but not in ours where we are creating a secure backoffice environment. This rebuild was happening every time the docker container started, and hence the geoserver environment was in an indeterminate state.

As a result we are making our own ;-)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.