5

We have GP tools in GP toolbox OurServices.tbx and some of them are written in Python. We access them via geoprocessing services. However some Python functions are just copy-pasted from one GP function script to another.

The question is: Is it possible to embed a Python script, say our_utils.py, into the toolbox file (tbx) so in all our Python tools we are able to access shared functions like this:

import our_utils
our_utils.sharedFunc()

We don't want pack shared functions as separate GP tools and then use ModelBuilder; this approach is too heavyweight for the issue.

Bounty edit 1 (2015-05-05)

I feel I have to give a bounty to someone participating here. At least three beautiful minds were attracted by it so I don't want my bounty going to dust.

I don't have ArcGIS at my fingerprints at the moment but I've spent 30 to 40 mins today checking if @mr.adams hacks can be extended to server installations (geoprocessing rest services). And, surprisingly, it worked with some additional hacks!

The hacks were: after packaging to .sd file (still a mystery for me how to pack more than one tool) rename that file to .zip and add libraries folders (import mylib => you must add mylib folder into zip). Then rename .zip back to .sd.

So my modest bounty goes to @mr.adams. I won't mark his answer as true answer; I still need to find a simpler way.

For pure Pythonistas @Farid_Cher gave an excellent .pyt template. I hope many copypasters will use it.

  • I'm unclear whether you are talking about Python script tools in a standard toolbox (*.tbx) or a Python toolbox (*.pyt) of tools. You can do import our_utils ... our_utils.doSomething() with both, but your question asks about embedding that leaves me wondering which you are using. Also, what version of ArcGIS for Desktop are you using? Can you edit your Question to revise it with these details, please? – PolyGeo Mar 1 '14 at 3:08
  • you should just need to sys.path.append('C:/Working/SomeFolder') before your import. If the linked Q doesn't answer your question please flag for reopening (and explain how it doesn't fit). – matt wilkie Mar 3 '14 at 17:57
  • @matt wilkie: I don't understand that "sys.path.append('C:/Working/SomeFolder') before your import". I want to import a .py file from my tbx, not from SomeFolder. It's more packaging issue... – Remigijus Pankevičius Mar 3 '14 at 19:14
  • ok, thanks for the clarification, re-opened. I think you'll likely have to export the py from the .tbx, so it sits as it's own file on disk. Or try converting it to a python-toolbox. – matt wilkie Mar 5 '14 at 17:50
  • 1
    In fact I look at future of .tbx as something like java jar file. It's a zip file, it has a file system structure (folders and files) so any class packed into jar can access other class using same notation as in file system. If you unpack jar into some folder, it works the same. Quite elegant. – Remigijus Pankevičius Mar 5 '14 at 18:23
1

Per @matt wilkie Suggestions, you should use python toolbox (.pyt) instead of a custom (.tbx) toolbox. However with python toolboxes you cannot have models. So if you have any models in your custom toolbox, then you have two options:

  1. to export those models to python script tools.
  2. to call tools in other toolboxes from within the Python toolbox code

Here is a template code for a python toolbox to suit your needs:

class our_utils: # Or put this class in a seperate file and import it here
    def sharedFunc():
        return

class Toolbox(object):
    def __init__(self):
        """Define the toolbox (the name of the toolbox is the name of the
        .pyt file)."""
        self.label = "Toolbox"
        self.alias = ""

        # List of tool classes associated with this toolbox
        self.tools = [Tool]


class Tool(object):
    def __init__(self):
        """Define the tool (tool name is the name of the class)."""
        self.label = "Tool"
        self.description = ""
        self.canRunInBackground = False

    def execute(self, parameters, messages):
        our_utils.sharedFunc()
        return

    ...

I hope it helps.

  • If .pyt is "add all functions into a single file" then I have already expressed my slightly negative opinions about this design here: gis.stackexchange.com/questions/88575/… – Remigijus Pankevičius May 3 '15 at 19:21
  • Your template is really good. Now I'm reading it and see: it has all what's needed when using Python toolbox: toolbox definition, definition of tools (adding one more tool: self.tools = [Tool1, Tool2]), and, of course, a simple way to add self-made shared utility functions. – Remigijus Pankevičius May 4 '15 at 18:58
  • 1
    In that link, you have mentioned ".pyt file is a valid python file however no Python IDE will recognize this fact". You can easily associate your IDE with this extension. For example If you use pyscripter, see this page support.esri.com/fr/knowledgebase/techarticles/detail/40258 – Farid Cheraghi May 4 '15 at 21:30
2

When you say embedding, I'm presuming you mean this approach? http://resources.arcgis.com/en/help/main/10.2/index.html#//005700000017000000

If so, short answer is no.

Embedding scripts is a nice way to minimize the number of files you have to pass around or just to hide/encrypt your work. You could embed your utilities as long as your utility functions are included in one of the tool's source scripts.

However, that embedded '.py' is now part of the toolbox, and Python's import can't peer into the toolbox to find it.

  • We don't want to hide anything :) – Remigijus Pankevičius May 3 '15 at 19:12
  • But you do want to embed, right? – DWynne May 4 '15 at 0:49
  • I want to pack, embed, whatever. We don't hide by encrypting. Users can read Python code if they want to. The main goal is to have one simple piece easy to install, upgrade to new version, etc. And I thought that tbx file can be such a piece. BTW, we use tbx because we have a mix of .NET and Python tools. – Remigijus Pankevičius May 4 '15 at 7:27
  • Ok, got it. So 'embedding' isn't a requirement per se, as it is a way to easily lay down and update/replace files. Have you looked at this approach: desktop.arcgis.com/en/desktop/latest/analyze/python/…. Might be a different way to achieve your goals. – DWynne May 4 '15 at 23:12
  • A good description is provided. This approach updates Python installation by installing "external libraries" (foo in the example). However it doesn't mention that 2 installations will be needed if we want to use the same tools in Desktop and Server. Say, for 10.2 they are at C:\Python27\ArcGIS10.2\Lib\site-packages (Desktop) and C:\Python27\ArcGISx6410.2\Lib\site-packages (Server). – Remigijus Pankevičius May 5 '15 at 8:56
2
+50

In the past I've had to distribute a toolbox (.tbx) with many tools, each with their own script but many of which shared the same functions. I believe this is the same issue you are dealing with. In my case I solved the packaging issue with this file structure:

top_folder/
    tools.tbx
    scripts/
        tool1.py
            module_name/
                __init__.py
                general.py
                    (inside of this file)
                    def upper_case(string):
                        return string.upper()

It works like this:

  1. In the top_folder directory sits the toolbox itself, tools.tbx and a directory called scripts.
  2. In tools.tbx, all tools reference their own .py file in the scripts directory (make sure the path is set to relative in the tool properties), like tool1.py.
  3. In addition to the normal import statements at the beginning of tool1.py (arcpy, os, etc.) you can import functions defined in general.py like so:

from module_name.general import upper_case

  1. Now you can call upper_case() anywhere in tool1.py, and you could do the same in any other .py file in the scripts directory.

This works best if you are able to distribute top_folder as a zip file. Because you have set all tool paths to relative, it doesn't matter where the top_folder sits on your computer, as long as the other files all are in order. Note that you need the __init__.py file (just make an empty text file) so that python will recognize module_name as something that can be imported.

You should be able to download my own implementation of this concept here. At the end of the CLI Toolbox/documentation/Using the CLI Toolbox.pdf document there is a more detailed explanation. It was my first stab at a real python implementation so no guarantees on best practices, unfortunately.

Hope that helps!

EDIT:

Thanks, I appreciate the nod.

Further note for anyone else interested in this: The other main advantage that was important to me with this schema is that any enterprising user can cut the module_name directory from its location here and paste it into the site-packages directory of their local Python installation. All the tools will function as if nothing has changed (because of the way the import statements are constructed) but now you can import module_name into any scripts that use the same Python installation.

I had originally intended to just make a python package that users would install, but realizing that A) those words wouldn't mean anything to a lot of people and B) on network systems I really don't know where Python would sit anyway, using this schema would be the best of both worlds.

  • Your approach relies on the belief that ArcGIS will unpack tool.py from tbx in the same folder as tbx sits on. Then "import from scripts" makes sense. However when publishing tbx to ArcGIS server one need to add it into service definition file (.sd). Zip file, actually. I wonder if it's possible to hack it to extract /srcipts folder before executing tool.py from -tbx. – Remigijus Pankevičius May 3 '15 at 19:18
  • 1
    Interesting. Yeah, I don't have any experience with publishing toolboxes, so I can't be much help there... Good luck though, this would be a nice thing to solve. – mr.adam May 3 '15 at 19:36
  • Your approach will be an interesting hack to try :) – Remigijus Pankevičius May 3 '15 at 19:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.