There's a great deal of information theory embedded in the management of GIS service queues. Much of it involves ugly math (summations involving exponential functions). I'm going to try to distill the theory to basics, then apply it to your question.
First off we need some definitions:
- Service - A general class of resource that provides work
- Instance - An individual worker for a service
- Pool - The number of available workers
- Request - A work unit that needs servicing
- Queue - The ordered list of requests awaiting servicing
- Job - A request being actively serviced by a service instance
- Runtime - The time is takes for an instance to service a job
- Limit - A resource constraint on an instance that can place a ceiling on runtime
- Exception - A failure in the processing of the request
Now to put them into common context:
You're due to fly out on an international flight. You have baggage and an economy ticket. You locate your airline, and join the group of folks waiting to check in. There are three slightly harried airline representatives beyond the eight people ahead of you, and two of the three have customers they're already checking in. As you wait, you notice that customers leave the front at an average of one minute per person, plus 45 seconds for each checked bag.
Under this content, when you first arrived you have a service (airline check-in) with three server instances (pool size three), eleven requests, with queue length nine, two active jobs, and a runtime proportional to the number of seats plus bags.
There are ways to speed you on your way:
The airline could hire a dozen more check-in specialists. This would provide more instances, and therefore more completed jobs per unit time. But the extra salary would increase costs to the point that fares tripled, and your holiday might fall victim.
The airline could limit two persons and three bags per job (which would also yield also more jobs/time) but you probably want to keep both your spouse and infant daughter on the flight with you (getting kicked out of the line would be a service exception, as would still being in the queue when your flight announced final boarding).
Let's say that there just happens to be an additional check-in specialist, who was on break. This individual walks up to an open podium, logs in with two-factor authentication, flips some switches to make the light turn on, and calls to the next person in line. Now you've experienced service pooling -- the worker was available, but it took a bit to spin up to become an instance.
In the GIS realm, the same principles can be applied, but let's describe the system based on the load you've described (assuming a one-min, two-max service list defaults):
You are running an airport with 100 airlines and 100 two-position check-in terminals, with 100 representatives at lit check-in stations, and 100 extra staffers in the back room to take on additional work (one per airline). We don't know how often a flyer comes in, or to which airline, or whether any representatives are busy, or how profitable the airport is that it can afford to have so many workers sitting around waiting for not enough customers. We do know that planes are leaving without passengers on occasion.
Would increasing the number of back-room staffers for one airline help prevent unhappy flyers? Well, maybe for that one airline, but in the context of the airport, probably not.
What you need to do is:
- Identify which services are consistently busy, and which are usually idle
- Reduce minimum pool from those that are often idle
- Add maximum pool size to the busiest services, highest queue length first (but add slowly, no more than half the backlog at a time; workers are expensive)
- If requests to certain servers are of short duration, and extremely bursty, it might make sense to increase the minimum number of instances, to reduce the latency of spinning up pooled instances
- Review and repeat as necessary
In the end, you only have so many resources, and it's not clear what the error might be. An expensive way to find out if your error is due to resource exhaustion is to double the capacity of the server (2x CPU and/or 2x RAM), or double the number of servers (the latter increases overhead cost, due to additional communication, but also provides a means to continue servicing during maintenance windows). The less expensive options all involve more work, since you'd need to do some research into queuing theory and review the practical aspects of your deployment in light of this theory. In between might be a short-term consult from a expert to review your configuration and make recommendations.
