There are a few geoprocesses on a recently restored ArcGIS Server 10.0 SP5 that are behaving strangely. If we turn on 'show messages' and try to run the process from the rest interface, the process will report 'failed due to crash or termination of the server object'. This happens at different points in the script and sometimes it will even return messages after the job has officially failed. Every now and then, the final message is 'the job has succeeded' even though the 'Job status' field has been set to 'failed' half way through.
Some information about the setup: Data and server directories are accessed via UNC path Authentication is handled via domain accounts Geoprocesses are backed by Python scripts
I've been searching around for file access issues in both the data store and server output directories. Any advice on best practices to narrow down what may be going on?