For several months our map services in our ArcGIS Server 10.5.1 setup occasionally fail to draw/render when browsing to their export REST endpoints.
Our setup is in the cloud and we use cluster setup. We also are using SDE. We have tested and verified that the issue is not because of any load or any service configuration (such as recycling time, etc.) or network issue or firewalls, etc. It just happens sporadically. We haven't been able to pinpoint the cause of this behaviour. All thise using HTTPS.
When browsing the REST endpoint of a published map service, a blank image is rendered sporadically. In other words, image gets displayed correctly sometimes and sometimes image is blank. So on the client side web interface, when user browse the map, a map service will render correctly and as soon as user pans, the map service will not display. Again when user pans the map, map service will render correctly. This happens at anytime (i.e. nothing fixed or no pattern observed).
We have observed that this happens to particular services and haven't seen such wierd behaviour for other map services.
Investigation carried so far:
- Restarting the bad map service in Server Manager or ArcCatalog.
- Restarting the ArcGIS Server Windows service.
- Restrting the cloud servers VM itself.
- Trying various service configurations such as recycling intervals, etc.
- Load testing and automatic querying the map service with many requests.
- Creating MXD from scratch and publishing it.
- Verifying MXD by using simple symbology or not using complex definition queries or complex labelling, etc.
- Checking data integrity.
- Verifying for reliable network.
- Verifying if one map service is failing to rendering then other services are working as expected.
- Increasing heap sizes.
- Checking AGS logs (no errors encountered).
Has anyone heard of or seen something similar to this? And has anyone resolved it or found the cause of this?