0

Googling this issue is getting me nowhere so I thought I'd cast a wide net here in the hope of getting an actionable lead on this.

Our GeoServer instance is configured for GeoWebCache. While troubleshooting a recent performance issue* (more on that at the bottom) I was combing thorough logs, and I found our GeoServer log (/usr/share/opengeo-suite-data/geoserver_data/logs/geoserver.log) to be loaded down with frequently-occurring errors like these:

2019-02-20 12:31:52,209 ERROR [seed.TruncateTask] - During truncate request: For input string: ""
2019-02-20 12:31:54,006 ERROR [seed.TruncateTask] - During truncate request: For input string: ""
2019-02-20 12:31:55,752 ERROR [seed.TruncateTask] - During truncate request: For input string: ""
2019-02-20 12:31:57,909 ERROR [seed.TruncateTask] - During truncate request: For input string: ""
2019-02-20 12:32:00,905 ERROR [seed.TruncateTask] - During truncate request: For input string: ""
2019-02-20 12:32:02,189 ERROR [seed.TruncateTask] - During truncate request: For input string: ""
2019-02-20 12:32:04,301 ERROR [seed.TruncateTask] - During truncate request: For input string: ""
2019-02-20 12:32:10,371 ERROR [seed.TruncateTask] - During truncate request: For input string: ""
2019-02-20 12:32:35,921 ERROR [seed.TruncateTask] - During truncate request: For input string: ""
2019-02-20 12:32:43,630 ERROR [seed.TruncateTask] - During truncate request: For input string: ""

As I understand it, the GWC Seeding and Truncating functionality is surfaced through a web-facing API, probably because GeoServer's GUI is web-facing in the first place, and can be managed programmatically with web requests using CURL, as discussed and demonstrated here.

However. As far as I know--and I'm pretty certain--we don't have any automation hitting GWC. Moreover, the tile layers we do have served by GeoServer/GWC are rather static, and the underlying data is infrequently replaced.

Per the documentation cited above, when I try checking the REST interface to "get the current state of seeding threads" by calling http://www.our.com/geoserver/gwc/rest/seed.json, it doesn't indicate any hanging/unfinished tasks. Specifically, it outputs:

long-array-array []

Trying to catch the ghost in the act, I decided to tail -f that log, and when I caught a blast of these errors coming in, I refreshed the seed.json endpoint, and again, it output no active seeding threads.

If I request 'http://www.our.com/geoserver/gwc/rest/seed/', which should show me any running tasks for the named layer (in this case, empty string), it throws a 500 error.

I see that the API exposes a method for terminating tasks, including a kill_all command, but the API is showing me I don't have any to terminate. The logs, though, are suggesting a different scenario.

Potentially interesting detail:

Looking at our GeoWebCache settings, I see the following in our disk quota configuration..

Set maximum tile cache size:
Used 70.01 GB out of 70.0 GB available.
When forcing disk quota limits, first remove tiles that are:

[x] Least frequently used

Does the fact that our GWC allocated quota is maxed out suggest the rampant seed.TruncateTask error is related to GeoServer's own quota management behaviors? The only ways I can think to test this would be to either 1. raise the quota, or 2. delete the tiles for a handful of less-popular layers to see if we quit seeing the error until we start spilling over the quota again.

On the contrary, I'm having trouble confirming this over-quota detail, as checking disk usage of my gwc directory shows me, curiously, one-half the utilization that GeoWebCache is claiming..

cd /usr/share/opengeo-suite-data/geoserver_data/gwc
du -hs
35G

How can I find source of all these calls to seed.TruncateTask, why are they submitting requests for empty string "", and/or how can I stop them?

Note that since these are actual error messages, I don't want to modify logging behaviors as a solution.

Experiment in progress

As an experiment, I have raised the disk quota from 70GB to 80GB, and I am tail -f'ing the log file for the rest of the day. These errors are frequent-enough that if this is not quota-related, then I should be seeing them soon enough. However, if I don't see any for the remainder of the day, I will feel pretty confident that this seed.TruncateTask error is caused by GWC's own disk-management functionality.

Obligatory stack details: Old Ubuntu, Old Apache, Old Tomcat, Old Geoserver.

*For anyone who lands here desperate for solutions to slow GeoServer performance, check the size of your tomcat log(s). In our case, the log /var/log/tomcat6/catalina.out had reached 2.1 GB and, apparently, it caused a trickle-down effect that dramatically hurt GeoServer's performance. While you're there, you might also want to check that directory for other burgeoning log files.

  • without knowing actual versions it's hard to tell - but this might be the fix github.com/GeoWebCache/geowebcache/commit/… (2014) – Ian Turton Feb 20 at 20:03
  • that string doesn't seem to exist in the current code base – Ian Turton Feb 20 at 20:05
  • @IanTurton, thanks for giving this some consideration. Ours is probably such an old version that I wouldn't be shocked if this is a bug that has been resolved by now. I can tell you this: I increased the quota and rebooted the system to make sure the new config fully propagated, and once it was back up, I started tail -f'ing the log file, and I haven't seen that error since. At the pace I was receiving it, barring a wild coincidence in usage patterns, I should have seen more of those by now. So it seems very likely related to cache quota management in our version of Geoserver/GWC. – elrobis Feb 20 at 20:53
  • It's a weird version, packaged as "opengeo-suite", and installed some time ago by my predecessor. Per the about page, it's: Version 2.1 SNAPSHOT, Build Date 08-Mar-2012, GeoTools Version 2.7 SNAPSHOT. – elrobis Feb 20 at 20:56
  • FWIW, @IanTurton, you've actually remarked on the impressive age of our render stack once before :) – elrobis Feb 20 at 20:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.