I'm in arcpy scripting for couple of years and after this time i've got a whole lot of scripts, packages and so on. The problems is that many of this scripts should be used over and over, many of them use the same logic (couple of point-to-point spatial joins with different limitations). Additional problem is that I reuse some part of this scripts: extentded arcpy.da.Cursor class for example (not only this one, it's about 5 classes and 5 modules with functions. So, 10 "scripts" in total)

Now the time to orginize this mess has came. At this moment I use project structure. One project - one toolset with it's own pack of scripts. I just copy from base location to project's one if I need my class in another project

Additional problem is that this "package" should be portable as easy as possible.

Almost all of this scripts/tools are used inside ArcMap environment.

Now I think of a structure like this:

_docs                              # sphinx-generated docs
gptools                            # main "toolbox"
    --- __init__.py                # access everything from outside and generate docs
    --- gptooset1                  # same logic script. Just as ArcMap toolboxes.
        --- __init__.py            # i could import tools from here
        --- gptooset1.tool1.py     # single tool
        --- gptooset1.tool2.py
    --- gptooset2
    --- ...
scripts                            # main wideused classes, functions, etc.
    --- __init__.py                # use it from outside
    --- class1.py
    --- ...
    --- functions1.py
    --- ...
site-packages                      # third-party modules, should be imported from here
    --- module1
toolboxes                          # python-toolboxes to access tools from ArcMap

There is few issues. First of all I can't import class1 to gptoolset1.tool1.py with relative import. There is seems to be only one way to do it through sys.path.append() which is

The second, the main one is that this structure is against Python basic structure.

    --- class1.py
    --- functions1.py

# tool1.py
from scripts import class1
from scripts import functions1

Does anyone has an experience of organizing such large packages? How to organize it well from Python point of view and keep it as simple is possible? (bulk of sys.path.appends would solve it but its awkward.)

UPD I don't think it's a general software question because it's ArcGIS specific. Python "software" uses up->down (complicated->easy) logic, when main modules are uses "small pieces of logic" by importing it. It gives a strict scheme (packages, subpackages and so on).

ArcGIS uses reversed logic: tools (python modules actually) are on the bottom of project structure and oftenly uses one common parts of code. If you put tools on top of your project you would get a mess of scripts or should copy-paste common parts from one location to anoter.

The question is how to combine this two completely different approaches. Why not SO? Because it's not that frequent case in python scripting but it is a part of ArcGIS environment.

closed as off-topic by BBG_GIS, aldo_tapia, whyzar, HDunn, MappaGnosis Nov 17 '17 at 9:11

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions relating to general IT or with no clear GIS component, are off-topic here but can be researched/asked at Stack Overflow (software development), Super User (computing hardware and software), Database Administrators (relational databases) and other SE sites" – BBG_GIS, aldo_tapia, whyzar, HDunn, MappaGnosis
If this question can be reworded to fit the rules in the help center, please edit the question.

  • @Sege Norin I think this topic is about python language architecture and structure.It's not about arcpy and arcgis. If you understand python architecture you can organize the packages – BBG_GIS Nov 16 '17 at 18:52
  • @wetland I don't think it's a general python question because Arcgis tools and arcpy itself are against python projects logic (because of reversed imports for example). So the question is how to find "the golden mean". – Serge Norin Nov 16 '17 at 20:12

To get around the relative import issue, I have started placing the modules that I've written in my interpreter Library (Lib) directory, that way they are able to be imported by any script that is using that interpreter.

  • It's a good way to avoid relative imports but it makes a real pain to port this package somewhere else. I should maintain two separate modules instead of one and teach my co-workers using python (how to setup and upgrade modules at least) – Serge Norin Nov 16 '17 at 21:11

Not the answer you're looking for? Browse other questions tagged or ask your own question.