I have a .pyt Python toolbox which runs fine in ArcMap and then gives the dreaded "ERROR 000816: The tool is not valid" when published as a GP Service. I've dealt with this issue many times before, but this one is exceptional.
If I run ArcMap as the arcgis service account user (Shift - Right click - Run as different user) and inspect the actual GP service code that ESRI creates, the .pyt is fine and runs with no issue, but the derived .tbx has red x's. I can't think of any reason why or any way to tell what the problem is.
The "extracted" .pyt also runs fine from the command line (run as arcgis service account user with debug code in the if __name__ == "__main__":
block. Running with 64 bit python and import arcserver at top of script).
This is on ArcGIS Server/Desktop 10.3.1.