0

I set up GeoServer (2.15) to run from a docker instance and mounted the data_dir externally. I then configured my server with a workspace, layers, styles and layer groups.

The docker instance was stopped and started as I resized my environment to have more memory. However, now GeoServer has lost some of the config when restarted.

I can see the workspace and styles have survived but the layers and layergroups have not. Clicking layergroups in the backend shows a page of Java errors.

Can I recover this situation by starting afresh and just copying in my workspace? I tried using the styles/workspaces/layergroups directories but as I said got the Java errors.

FINAL EDIT

OK a significant edit to the question as it appears the problem is linked to an environment change.

I can reproduce as follows:-

  • create Amazon AWS EC2 instance
  • setup clean docker instance and install geoserver/geocache within instance, mount data_dir externally
  • configure geoserver via GUI
  • shutdown docker instance [sudo docker stop --time=60 $(sudo docker ps -q);]
  • resize EC2 instance (as a result the IP address may change)
  • restart docker instance [sudo docker start ]

Now Geoserver will not start but Tomcat will. I note the following errors in catalina.2019-05-09.log

INFO [java-sdk-http-connection-reaper] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [org.apache.http.pool.AbstractConnPool$3]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
 java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.http.pool.AbstractConnPool$3]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1372)
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1360)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1219)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1180)
    at org.apache.http.pool.AbstractConnPool.closeIdle(AbstractConnPool.java:622)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.closeIdleConnections(PoolingHttpClientConnectionManager.java:423)
    at com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:172)

So I restarted Tomcat and now Geoserver "almost runs" but dies with these errors:-

org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [geoserver] appears to have started a thread named [FileSystemWatcher-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 java.lang.Thread.run(Thread.java:748)
09-May-2019 09:05:11.896 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [geoserver] appears to have started a thread named [java-sdk-http-connection-reaper] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Thread.sleep(Native Method)
 com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:157)
09-May-2019 09:05:11.897 SEVERE [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [geoserver] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@18757ee0]) and a value of type [org.springframework.security.core.context.SecurityContextImpl] (value [org.springframework.security.core.context.SecurityContextImpl@ffffffff: Null authentication]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
09-May-2019 09:05:11.898 SEVERE [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [geoserver] created a ThreadLocal with key of type [java.lang.ThreadLocal.SuppliedThreadLocal] (value [java.lang.ThreadLocal$SuppliedThreadLocal@5d2b5543]) and a value of type [org.geowebcache.storage.CompositeBlobStore.StoreSuitabilityCheck] (value [EXISTING]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
09-May-2019 09:05:11.903 INFO [localhost-startStop-1] java.util.prefs.FileSystemPreferences$1.run Created user preferences directory.
09-May-2019 09:05:11.915 SEVERE [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [geoserver] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@6e1ab278]) and a value of type [org.geotools.referencing.operation.projection.TransverseMercator.Provider] (value [PROJECTION["Transverse_Mercator"]]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
09-May-2019 09:05:11.916 SEVERE [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [geoserver] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3f670f47]) and a value of type [org.geotools.referencing.wkt.Formatter] (value []) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
09-May-2019 09:05:11.928 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /usr/local/tomcat/webapps/geoserver has finished in 73,038 ms
09-May-2019 09:05:11.933 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-apr-8080"]
09-May-2019 09:05:11.947 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-apr-8009"]
09-May-2019 09:05:11.951 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 74219 ms
09-May-2019 09:05:30.953 INFO [java-sdk-http-connection-reaper] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [org.apache.http.pool.AbstractConnPool$3]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
 java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.http.pool.AbstractConnPool$3]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1372)
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1360)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1219)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1180)
    at org.apache.http.pool.AbstractConnPool.closeIdle(AbstractConnPool.java:622)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.closeIdleConnections(PoolingHttpClientConnectionManager.java:423)
    at com.amazonaws.http.IdleConnectionReaper.run(IdleConnectionReaper.java:172)
Improve this question
5
  • just restore the data directory from your backup and be more careful stopping docker in future.
    – Ian Turton ♦
    May 8, 2019 at 8:31
  • All very well in hindsight but we were experimenting rather than creating a production instance. Copying the data from the backup dir does not work as I stated above. The bit that I'm puzzled by is that creating a new geoserver instance and copying in workspaces dir only restores the workspace and styles. Why are the layers and stores not copied in?
    – CitizenFish
    May 8, 2019 at 8:55
  • You need to copy all of the data dir in not just bits and peices
    – Ian Turton ♦
    May 8, 2019 at 9:11
  • Unfortunately that is exactly what breaks it all.
    – CitizenFish
    May 8, 2019 at 10:15
  • then your back up is corrupt, try with a clean install
    – Ian Turton ♦
    May 8, 2019 at 12:18

1 Answer 1

Reset to default
1

OK after a lot of frustrated digging around this turned out to be an issue with the docker build not the geoserver configuration.

Leaving this here just in case anyone uses the image https://github.com/kartoza/docker-geoserver

The image uses a startup script to rebuild which requires internet access, fine in 99.99% of cases but not in ours where we are creating a secure backoffice environment. This rebuild was happening every time the docker container started, and hence the geoserver environment was in an indeterminate state.

As a result we are making our own ;-)

Improve this answer
3
  • 1
    If you use the pre packaged docker images you will never have this . The entrypoint script only spins up tomcat and is not responsible for building the image
    – kartoza-geek
    Apr 26, 2020 at 14:14
  • @CitizenFish, Can you please pin point what was the issue that you faced when building the docker image , We are facing the exact same issue now.
    – Viswas Menon
    Apr 6, 2023 at 1:41
  • We didn't get any further than noting that the startup script required internet access as above. Apologies but we abandoned this approach
    – CitizenFish
    Apr 14, 2023 at 8:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.