ArcGIS server Geoprocessing job failed immediately after submit

Question

I have a few geoprocessing services published onto a production ArcGIS server but after a series of Windows 2012R2 and ArcGIS server updates, the services stopped function.

When I submit a job, these are the messages I would get:

• esriJobMessageTypeInformative: Submitted.
• esriJobMessageTypeInformative: Executing...
• esriJobMessageTypeError:
• esriJobMessageTypeError: Failed.

To test and see if there is something wrong with the server, I published an empty python script. When I submit a job to that service, I get the same set of message. If it had function normally, there wouldn't be any "esriJobMessageTypeError". Its as if the code did not execute at all, stopped at the first line.

Does anyone have similar problems after updates and any fixes?

---

UPDATE
Attempting to built my service from scratch step by step and I think I have located where the problem started.

Starting from the beginning,

1. Published an empty python script to a test python service and confirmed that it is running normally.
2. Update the script with some imports. Service is executing normally.
3. Update the service with one input parameter. The service failed with the above 4 lines of esriJobMessag. However server log did not record anything

---

UPDATE
Have test the empty script with a single parameter on another ArcGIS server and the server execute the service with no problem.

So what should I do next? Anything before completely re-install the problem ArcGIS server?

Answer 1

I had a similar problem, but I don't think it happened from the same reason, however, adding a relatively general solution for anyone else stumbling here.

My problem started happening after an ArcGIS Server update to version 10.61, it happened because my python script used a *.mxd file that was in a folder that wasn't registered as a data source.

So my script ran locally but failed instantly when used as a service.

I found the problem using two logging solutions:

1. the service log which even when instructed to log at the INFO level only logged errors. Inside the relevant log I found the python stack trace.
   You can find the services logs in your server machine (under Windows) at C:\arcgisserver\logs\{server name}\services\.
   There you have to navigate to the directory of the specific service and find the most recently updated log file. I found there that my jobs failed because the server could not find the *.mxd file that the script had to use.

2. After I knew where my script failed, I added custom logging using the logging module. which helped me find where the server was looking for the *.mxd (in a folder that doesn't exist.

This is a basic configuration for logging inside the script:

import logging
logging.basicConfig(filename=os.path.join('D:/{whatever folder you want logs in}','{log file name.txt/.log}'), 
                level=logging.INFO, 
                format='%(asctime)s %(levelname)s %(name)s %(message)s')
logger=logging.getLogger(__name__)

and every where you would want to log an INFO level message, just add:

logging.info('your message')

This was my resulting log and how I found that the server used the wrong path.
line 2 is from a local script activation through ArcMap, line 4 is when used through the gpservice.

I hope this will help anyone in logging your gp scripts.