0

If I understand correctly, pooling an AGS Service means that when a client connects to a service, an instance is created and an ArcSOC process is executed. Setting minimum to 0 thus doesn't start the service until a call is made to the server to start the service. The first call is therefore a bit slower due to the service having to first start. What I am not sure about though, is the parameter "maximum" number of services. Default is min-1 and max-2.

But what if I raised the maximum number to 24?

Would that mean that I could create up to 24 ArcSOC Processes on my server which could in theory produce bottlenecks?

Lately I have been experiencing service crashes with raster services, i.e. either MapServices which serve rasters or Imageservices. I have 32GB Ram on a windows 10 machine and around 100 Services running constantly with standard pooling (min:1 Max:2).

If my CPU and RAM usage is generally not higher than 50% can I assume everything is running fine?

1
  • 1
    You've got a lot of questions here. Unfortunately, there isn't much that is GIS-centric to the queuing theory which is confounding you (the same principles apply to banks, supermarkets, and airline checkins). The answer to your final question is "Certainly not," because you've been experiencing service failures. 32GB may be low for a GIS server (my gaming PC is at 64GB), and 100 inactive services is probably far too many (especially if the host is light on CPUs as well). Esri has training available on tuning ArcGIS Server, in addition to documentation and whitepapers.
    – Vince
    Feb 28, 2021 at 15:40

2 Answers 2

3

There's a great deal of information theory embedded in the management of GIS service queues. Much of it involves ugly math (summations involving exponential functions). I'm going to try to distill the theory to basics, then apply it to your question.

First off we need some definitions:

  • Service - A general class of resource that provides work
  • Instance - An individual worker for a service
  • Pool - The number of available workers
  • Request - A work unit that needs servicing
  • Queue - The ordered list of requests awaiting servicing
  • Job - A request being actively serviced by a service instance
  • Runtime - The time is takes for an instance to service a job
  • Limit - A resource constraint on an instance that can place a ceiling on runtime
  • Exception - A failure in the processing of the request

Now to put them into common context:

You're due to fly out on an international flight. You have baggage and an economy ticket. You locate your airline, and join the group of folks waiting to check in. There are three slightly harried airline representatives beyond the eight people ahead of you, and two of the three have customers they're already checking in. As you wait, you notice that customers leave the front at an average of one minute per person, plus 45 seconds for each checked bag.

Under this content, when you first arrived you have a service (airline check-in) with three server instances (pool size three), eleven requests, with queue length nine, two active jobs, and a runtime proportional to the number of seats plus bags.

There are ways to speed you on your way:

  • The airline could hire a dozen more check-in specialists. This would provide more instances, and therefore more completed jobs per unit time. But the extra salary would increase costs to the point that fares tripled, and your holiday might fall victim.

  • The airline could limit two persons and three bags per job (which would also yield also more jobs/time) but you probably want to keep both your spouse and infant daughter on the flight with you (getting kicked out of the line would be a service exception, as would still being in the queue when your flight announced final boarding).

Let's say that there just happens to be an additional check-in specialist, who was on break. This individual walks up to an open podium, logs in with two-factor authentication, flips some switches to make the light turn on, and calls to the next person in line. Now you've experienced service pooling -- the worker was available, but it took a bit to spin up to become an instance.

In the GIS realm, the same principles can be applied, but let's describe the system based on the load you've described (assuming a one-min, two-max service list defaults):

You are running an airport with 100 airlines and 100 two-position check-in terminals, with 100 representatives at lit check-in stations, and 100 extra staffers in the back room to take on additional work (one per airline). We don't know how often a flyer comes in, or to which airline, or whether any representatives are busy, or how profitable the airport is that it can afford to have so many workers sitting around waiting for not enough customers. We do know that planes are leaving without passengers on occasion.

Would increasing the number of back-room staffers for one airline help prevent unhappy flyers? Well, maybe for that one airline, but in the context of the airport, probably not.

What you need to do is:

  • Identify which services are consistently busy, and which are usually idle
  • Reduce minimum pool from those that are often idle
  • Add maximum pool size to the busiest services, highest queue length first (but add slowly, no more than half the backlog at a time; workers are expensive)
  • If requests to certain servers are of short duration, and extremely bursty, it might make sense to increase the minimum number of instances, to reduce the latency of spinning up pooled instances
  • Review and repeat as necessary

In the end, you only have so many resources, and it's not clear what the error might be. An expensive way to find out if your error is due to resource exhaustion is to double the capacity of the server (2x CPU and/or 2x RAM), or double the number of servers (the latter increases overhead cost, due to additional communication, but also provides a means to continue servicing during maintenance windows). The less expensive options all involve more work, since you'd need to do some research into queuing theory and review the practical aspects of your deployment in light of this theory. In between might be a short-term consult from a expert to review your configuration and make recommendations.

2

Have you read https://enterprise.arcgis.com/en/server/latest/publish-services/windows/tuning-and-configuring-services.htm?

If you set the pool size into 24 and you have enough users to start them all then you will have 24 instances up. If load reaches 50% with max 2 instances then 24 instances will probably allow your users to start so many processes that it will kill your server. But you might have enough resources for 3 or 4 instances. That could cut down the queuing time of your users at the most busy periods.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.