Python toolbox documentation from within pyt file

Question

Python toolboxes are a great way to create custom, GUI toolboxes for ArcGIS without actually interacting with the GUI (since everything can be defined within the pyt file). Regrettably (and incomprehensibly) documenting a python toolbox does require the GUI (see Documenting a tool in a Python toolbox). This help page also states:

For Python toolboxes, the documentation for the toolbox and tools are stored in .xml files that are associated with the toolbox and tools by name.

In a question on this topic the user @dmahr stated in his answer:

Fortunately, Python supports for reading and writing XML, so you should be able to dynamically edit the help text from a separate script.

How would something like this be achieved? Since my toolbox is constantly changing, I'd like code and documentation to live in the same file.

EDIT:

As @user2856 commented, the first step would be to look at the xml file generated when documenting the tool via the GUI. This step should have been part of the question, so here's the xml for the tool named "Tool" (saved with the name New Python Toolbox.Tool.pyt.xml:

<?xml version="1.0"?>
  -<metadata xml:lang="de">
    -<Esri>
      <CreaDate>20190401</CreaDate>
      <CreaTime>10300800</CreaTime>
      <ArcGISFormat>1.0</ArcGISFormat>
      <SyncOnce>TRUE</SyncOnce>
      <ModDate>20190401</ModDate>
      <ModTime>14352000</ModTime>
      -<scaleRange>
      <minScale>150000000</minScale>
      <maxScale>5000</maxScale>
      </scaleRange>
    </Esri>
    -<tool xmlns="" toolboxalias="" displayname="Tool" name="Tool">
      <arcToolboxHelpPath>c:\program files\arcgis\pro\Resources\Help\gp</arcToolboxHelpPath>
      -<parameters>
      -<param displayname="Input Features" name="in_features" expression="in_features" datatype="Feature Layer" direction="Input" type="Required">
          <dialogReference><DIV STYLE="text-align:Left;"><DIV><P><SPAN>this is the in_features dialog explenation </SPAN></P></DIV></DIV></dialogReference>
          <pythonReference><DIV STYLE="text-align:Left;"><DIV><P><SPAN>and here is the scripting explenation</SPAN></P></DIV></DIV></pythonReference>
        </param>
      </parameters>
      <summary><DIV STYLE="text-align:Left;"><DIV><P><SPAN>This is a nice summary</SPAN></P></DIV></DIV></summary>
      <usage><DIV STYLE="text-align:Left;"><DIV><P><SPAN>This is how this tool should be used</SPAN></P></DIV></DIV></usage>
    </tool>
    -<dataIdInfo>
      -<idCitation>
          <resTitle>Tool</resTitle>
        </idCitation>
        -<searchKeys>
          <keyword>Here are some tags</keyword>
        </searchKeys>
    </dataIdInfo>
    -<distInfo>
      -<distributor>
        -<distorFormat>
          <formatName>ArcToolbox Tool</formatName>
        </distorFormat>
      </distributor>
    </distInfo>
    -<mdHrLv>
      <ScopeCd value="005"/>
    </mdHrLv>
    <mdDateSt Sync="TRUE">20190401</mdDateSt>
  </metadata>

Answer 1

This was bothering me probably more than it should have, and after getting to know python a little bit I've found that the problem is pretty easy to solve. Since I'm mostly an R user, these lines of python code might not be very pythonic. Feel free to comment / edit:

EDIT: Since the whole script is fairly long I've added everything to a Github Repo: https://github.com/ratnanil/PythonToolboxDocumentation

Step 1: Adding attributes within .pyt File

Add the documentation to the .pyt File as attributes to the objects. For example, adding adding a "Dialog Reference" for an individual Parameter can be created like so:

param0 = arcpy.Parameter(
    displayName="Input workspace",
    name="in_workspace",
    datatype="DEWorkspace",
    parameterType="Required",
    direction="Input")

param0.dialogref = "Enter explanatory information about this tool here"

You are free to choose your own names for the attributes (eg dialogref), just make sure you are not overwriting existing names (name, direction, value etc.).

Step 2: Create a python Script to extract the tool's documentation and write xml Files

This file will hold the code to extract all the documentation from the .pyt File and write the xml files based on this information.

import shutil
import importlib
import os
import datetime
import xml.etree.cElementTree as ET

# enter the filename of pyt-File WITHOUT extension
pythontoolbox = "mytoolbox" 

# this function copies the .pyt- to a .py-file (and deletes a previous version if it exists). We need this because I have not found a way to directly import a .pyt-file
def copypyt(pythontoolbox):
    pyfile = pythontoolbox+".py"
    pytfile = pythontoolbox + ".pyt"
    if os.path.isfile(pyfile):
        os.remove(pyfile)
    shutil.copy(pytfile, pyfile)

# run the above function to create a .py Copy of the .pyt-File
copypyt(pythontoolbox)

# imports the toolbox based on the "pythontoolbox"-Variable. The module is first stored in a variable for later use
import_toolbox = importlib.import_module(pythontoolbox)
import_toolbox

# since the module is stored in a variable, we can reload it in case the pyt file experienced changes during the session
importlib.reload(import_toolbox)

# import all the tools from the toolbox
toolbox_tools = import_toolbox.Toolbox().tools
# we could also import the toolbox label and alias, but we don't need in this minimal example
# toolbox_label = import_toolbox.Toolbox().label
# toolbox_alias = import_toolbox.Toolbox().alias

# Get today's date 
today = datetime.datetime.now().strftime("%Y%m%d")

# the following for-loop creates an .xml file for each tool, including all metadata extracted from the pyt file
for tool_i in toolbox_tools:

    # creates the first level tag named "metadata"
    metadata = ET.Element("metadata")
    # creates a sublevel tag named "esri"
    esri = ET.SubElement(metadata, "Esri")

    # Add todays date to the xml Tag "CreaDate", but maybe the creation date of pyt would be more appropriate
    ET.SubElement(esri, "CreaDate").text = today

    # I don't care too much about some of the elements in the xml file. In this example, I've included them none the less but filled with some dummy data. these are marked with a # *
    ET.SubElement(esri, "CreaTime").text = today             # *
    ET.SubElement(esri, "ArcGISFormat").text = "1.0"         # *
    ET.SubElement(esri, "SyncOnce").text = "TRUE"            # *
    ET.SubElement(esri, "ModDate").text = today              # *
    ET.SubElement(esri, "ModTime").text = today              # *
    scaleRange = ET.SubElement(esri, "scaleRange")
    ET.SubElement(scaleRange, "minScale").text = "150000000" # *
    ET.SubElement(scaleRange, "maxScale").text = "5000"

    tool_name = tool_i.__name__
    tool_fullname = str(tool_i).replace("<class '", "").replace("'>", "")
    tool_label = tool_i().label
    tool_description = tool_i().description
    tool_canRunInBackground = tool_i().canRunInBackground = False
    tool_category = tool_i().category

    tool = ET.SubElement(metadata,
                         "tool",
                         xmlns="",
                         name=tool_name,
                         displayname=tool_label,
                         toolboxalias="")

    ET.SubElement(tool,"arcToolboxHelpPath").text = "c:\\program files\\arcgis\\pro\\Resources\\Help\\gp" # *

    parameters = ET.SubElement(tool, "parameters")

    # this nested loop writes the documentation for all parameters
    for param_i in tool_i().parameters:
        param_displayName = param_i.displayName
        param_name = param_i.name
        param_datatype = param_i.datatype
        param_parameterType = param_i.parameterType
        param_direction = param_i.direction
        param_value = param_i.value

        # For all custom parameters, include a failsafe in case some attributes were note defined for some tools
        hasattr(toolbox_tools[1]().parameters[1],"pythonref")
        if hasattr(param_i, "dialogref"):
            dialogref = param_i.dialogref
        else:
            dialogref = ""
        if hasattr(param_i, "pythonref"):
            pythonref = param_i.pythonref
        else:
            pythonref = ""

        param = ET.SubElement(parameters,
                              "param",
                              name=param_name,
                              displayname=param_displayName,
                              type=param_parameterType,
                              direction=param_direction,
                              datatype=param_datatype,
                              expression=param_name)

        reference_style_before = "<DIV STYLE=\"text-align:Left;\"><DIV><P><SPAN>"
        reference_style_after= "</SPAN></P></DIV></DIV>"
        dialogReference = ET.SubElement(param, "dialogReference").text = reference_style_before + dialogref + reference_style_after
        pythonReference = ET.SubElement(param, "pythonReference").text = reference_style_before + pythonref + reference_style_after

    summary = ET.SubElement(tool,"summary").text = reference_style_before + tool_description + reference_style_after
    usage = ET.SubElement(tool,"usage").text = reference_style_before + "usage" + reference_style_after
    scriptExamples = ET.SubElement(tool,"scriptExamples")
    scriptExample = ET.SubElement(scriptExamples,"scriptExample")
    ET.SubElement(scriptExample,"title").text = "title for eg 1"
    ET.SubElement(scriptExample,"para").text = "some description for code sample 1"
    ET.SubElement(scriptExample,"code").text = "code sample 1"

    dataIdInfo = ET.SubElement(metadata, "dataIdInfo")
    idCitation = ET.SubElement(dataIdInfo, "idCitation")
    resTitle = ET.SubElement(idCitation, "resTitle").text = tool_label

    idCredit = ET.SubElement(dataIdInfo, "idCredit").text = "Credit whom credit is due" # *

    searchKeys = ET.SubElement(dataIdInfo, "searchKeys") 
    ET.SubElement(searchKeys, "keyword").text = "tag1, tag2, tag3" # *

    distInfo = ET.SubElement(metadata, "distInfo") # *
    distributor = ET.SubElement(distInfo, "distributor") # *
    distorFormat = ET.SubElement(distributor, "distorFormat") # *
    ET.SubElement(distorFormat, "formatName").text = "ArcToolbox Tool"

    mdHrLv = ET.SubElement(metadata, "mdHrLv")
    ET.SubElement(mdHrLv, "ScopeCd ", value = "005") # *
    mdDateSt = ET.SubElement(metadata, "mdDateSt",Sync="TRUE").text = today # *

    tree = ET.ElementTree(metadata)
    tree.write(tool_fullname+".pyt.xml")

Step 3: Create a python Script to extract the toolbox's documentation and write xml Files

Now you have created the most important part of the documentation (for each tool). In a similar manner, you can now create the documentation for the toolbox itself.

Answer 2

I know of no other way to create the documentation for parameters other than editing the Item Description accessible through ArcMap/ArcCatalog.

I think the poster in the other thread stating you should be able to dynamically edit was implying that you could edit the Item Description in a programmatic way as it is nothing more than an xml file. This of course would rely on you writing hundreds of lines of code and having a deep understanding on the metadata standards that ESRI are using.

As you state your code is constantly changing I would recommend holding off on describing the parameters until that particular tool becomes stable then document it. I say this as for example you create a parameter that is of type FeatureClass, you add a whole load of description to it, then later you decide to change it to type FeatureLayer (because lets says that makes your code easier to understand/ run better). By simply changing the type you will blitz all the description you had already added and you have to start all over again. I have fallen foul to this before so I document when the tool is stable and unlikely to change.