Optimising map cache creation performance in ArcGIS for Server 10.2.1

Question

I'm relatively new to ArcGIS for Server, so I hope that someone is able to point me in the right direction in case what I've been doing isn't good practice.

I have 2 boxes with ArcGIS for server 10.2.1, both on the same site. Both boxes have 4 processors and 16gb RAM. Both boxes run on Windows Server 2008.

The site is used to both provide a couple of base map services to a small amount of users (<5) and to generate cache tiles for future services.

I am currently generating cache tiles for a mapping service (~50GB). I would have expected to see the CPU usage on the 2 boxes running quite high. But it tends to be sitting between 15% and 30% on each box.

The maximum instances for the caching tools is set to 6.

The maximum number of instances per machine is set to 3.

Am I wrong in assuming that I should be seeing a higher usage of the CPU?

Have I not put in the correct figures?

Or is my set-up not best practice? i.e. should I be using one site just for serving maps and another site just for caching?

I think I've followed the guidelines mentioned here and here. But I'm pretty sure that the caching is running slower than it should. After 19 hours it has only cached 1.17% of all my tiles.

Any best practice suggestions much welcome.

UPDATE: After 21 hours the CPU usage on both machines is down to nothing:

machine 1:

machine 2:

The cache status "in progress" bar on the server is still moving, but the cache % hasn't increased in the last 2 hours.

Answer 1

It looks like you've done a pretty good job of following best practices for creating the cache. Your servers have enough horsepower, but pulling the map data from your database might be an issue. Here's a little summary of this site that has a few additional tips for getting the most bang for your buck.

1 - Analyze your map before publishing it!

This may be obvious, but I've often been too quick to publish to the server without checking the results from analyzing. Just go to File->Analyze Map and do a quick check to see if your map has any issues. The faster your map can render, the faster it can cache.

2 - Keep the data local

If you have a single machine deployment, keep the map data in a FGDB local to the server. If you have multiple machines, let each machine have a copy of the data and use the "Use local cache directory when generating tiles on the server" option when setting up the cache.

The link above has some handy tips on dealing with failures, along with this handy script that parses caching errors into a polygon footprint so you can easily go back and try to re-cache the failed areas.

This question comes up every now and again. Maybe we can get some more answers and turn this into a Wiki. :)

Answer 2

Another thing you could do is set the caching controllers and the caching tools to a different cluster. This should isolate the caching process from the regular pool of processes. I have seen this yield better performance isolating caching from regular arcgisserver functions. I typically call this new cluster caching. Then when you add new machines to your installation just add them to the caching cluster versus default and this will increase your processing power. Additionally, if you see low utilization then keep adding more and more resources until you get to around 70-80% utilization you still want to leave some horsepower to copy the files back and forth and reduce contention for resources.

Answer 3

Based on the ESRI documenation (http://server.arcgis.com/en/server/latest/publish-services/linux/accelerating-map-cache-creation.htm) the best practice for the number of instances to use for caching is n + 1 where n is the number of cores running on the server. Your servers have 4 processors each and each X5690 processor has 6 cores according to the documentation:

http://ark.intel.com/products/52576/Intel-Xeon-Processor-X5690-12M-Cache-3_46-GHz-6_40-GTs-Intel-QPI

Based on this you should theoretically be able to handle 25 caching instances per server. Now I know the ESRI document is for 10.3 but this has been the recommended practice for the last few versions.

I have a similar server setup and I can tell you that my CPU would get to 100% way before I would get close to 25 instances but I can use 10-15 quite comfortably.

So if you have tested that disk IO or the database are not the bottlenecks I would recommend increasing the number of instances to 10. If this doesn't increase your CPU usage it means it was not the limiting factor. If it does increase it but it's still reasonable you could try increasing further.

Anyway, moral of the story is don't be shy about cranking the instances. Once I also worked on a fairly underpowered server with only 2 cores and 3 instances got me nowhere. However when I increased the instances to 5 the CPU usage was reasonable. The best practice is just a starting point.