Is it possible to add python util scripts as shared libraries to esri geoprocessing toolbox?

Question

We have GP tools in GP toolbox OurServices.tbx and some of them are written in Python. We access them via geoprocessing services. However some Python functions are just copy-pasted from one GP function script to another.

The question is: Is it possible to embed a Python script, say our_utils.py, into the toolbox file (tbx) so in all our Python tools we are able to access shared functions like this:

import our_utils
our_utils.sharedFunc()

We don't want pack shared functions as separate GP tools and then use ModelBuilder; this approach is too heavyweight for the issue.

Bounty edit 1 (2015-05-05)

I feel I have to give a bounty to someone participating here. At least three beautiful minds were attracted by it so I don't want my bounty going to dust.

I don't have ArcGIS at my fingerprints at the moment but I've spent 30 to 40 mins today checking if @mr.adams hacks can be extended to server installations (geoprocessing rest services). And, surprisingly, it worked with some additional hacks!

The hacks were: after packaging to .sd file (still a mystery for me how to pack more than one tool) rename that file to .zip and add libraries folders (import mylib => you must add mylib folder into zip). Then rename .zip back to .sd.

So my modest bounty goes to @mr.adams. I won't mark his answer as true answer; I still need to find a simpler way.

For pure Pythonistas @Farid_Cher gave an excellent .pyt template. I hope many copypasters will use it.

Answer 1

When you say embedding, I'm presuming you mean this approach? http://resources.arcgis.com/en/help/main/10.2/index.html#//005700000017000000

If so, short answer is no.

Embedding scripts is a nice way to minimize the number of files you have to pass around or just to hide/encrypt your work. You could embed your utilities as long as your utility functions are included in one of the tool's source scripts.

However, that embedded '.py' is now part of the toolbox, and Python's import can't peer into the toolbox to find it.

Answer 2

In the past I've had to distribute a toolbox (.tbx) with many tools, each with their own script but many of which shared the same functions. I believe this is the same issue you are dealing with. In my case I solved the packaging issue with this file structure:

top_folder/
    tools.tbx
    scripts/
        tool1.py
            module_name/
                __init__.py
                general.py
                    (inside of this file)
                    def upper_case(string):
                        return string.upper()

It works like this:

1. In the top_folder directory sits the toolbox itself, tools.tbx and a directory called scripts.
2. In tools.tbx, all tools reference their own .py file in the scripts directory (make sure the path is set to relative in the tool properties), like tool1.py.
3. In addition to the normal import statements at the beginning of tool1.py (arcpy, os, etc.) you can import functions defined in general.py like so:

from module_name.general import upper_case

4. Now you can call upper_case() anywhere in tool1.py, and you could do the same in any other .py file in the scripts directory.

This works best if you are able to distribute top_folder as a zip file. Because you have set all tool paths to relative, it doesn't matter where the top_folder sits on your computer, as long as the other files all are in order. Note that you need the __init__.py file (just make an empty text file) so that python will recognize module_name as something that can be imported.

You should be able to download my own implementation of this concept here. At the end of the CLI Toolbox/documentation/Using the CLI Toolbox.pdf document there is a more detailed explanation. It was my first stab at a real python implementation so no guarantees on best practices, unfortunately.

Hope that helps!

EDIT:

Thanks, I appreciate the nod.

Further note for anyone else interested in this: The other main advantage that was important to me with this schema is that any enterprising user can cut the module_name directory from its location here and paste it into the site-packages directory of their local Python installation. All the tools will function as if nothing has changed (because of the way the import statements are constructed) but now you can import module_name into any scripts that use the same Python installation.

I had originally intended to just make a python package that users would install, but realizing that A) those words wouldn't mean anything to a lot of people and B) on network systems I really don't know where Python would sit anyway, using this schema would be the best of both worlds.

Answer 3

Per @matt wilkie Suggestions, you should use python toolbox (.pyt) instead of a custom (.tbx) toolbox. However with python toolboxes you cannot have models. So if you have any models in your custom toolbox, then you have two options:

1. to export those models to python script tools.
2. to call tools in other toolboxes from within the Python toolbox code

Here is a template code for a python toolbox to suit your needs:

class our_utils: # Or put this class in a seperate file and import it here
    def sharedFunc():
        return

class Toolbox(object):
    def __init__(self):
        """Define the toolbox (the name of the toolbox is the name of the
        .pyt file)."""
        self.label = "Toolbox"
        self.alias = ""

        # List of tool classes associated with this toolbox
        self.tools = [Tool]


class Tool(object):
    def __init__(self):
        """Define the tool (tool name is the name of the class)."""
        self.label = "Tool"
        self.description = ""
        self.canRunInBackground = False

    def execute(self, parameters, messages):
        our_utils.sharedFunc()
        return

    ...

I hope it helps.