Compiling Python scripts (to .exe) that use ArcGIS Geoprocessing Tools?

Question

I've been coding with Python for several months now and have developed some reasonably complex scripts for primarily geoprocessing tasks. That being said, I'm still learning a lot as I'm coming from a SQL/VBA/VBScript background.

I know that compiled code typically runs faster than code that must be processed by a language interpreter, so I'm interested in the possibility of compiling a geoprocessing Python script to a .EXE file for working with big data.

Is this even possible? If it is, what is the best way to compile a Python (.py) script that is importing the arcgisscripting or arcpy modules?

I spent a few minutes trying to find what I want to do and the search returned this article among others: http://www.ehow.com/how_2091641_compile-python-code.html

The compiler seemed to work, but upon executing the resulting .EXE file, it gave a cryptic error saying some files were unavailable.

The Python script runs what seems to be reasonably well from the command line, but I'm wondering if I could see some slight improvement if I were able to compile the .py file. Again, I'm working with some big datasets that are taking +20 hours to process (delineating watersheds from input water-quality sample sites). I'll take anything I can get in way of improvements.

The script ran 10% quicker outside of ArcGIS from the command line using a test set of sites versus setting the script up as a script tool in a new toolbox in ArcCatalog. I've been running the script from the command line w/o any instance of ArcGIS open on a dedicated machine.

So, is it possible to compile Python scripts that import the arcgisscripting module and that call ArcToolBox tools?

EDIT

Thanks for the input, this is helpful for me. The script is largely a way to coordinate a number of ArcGIS tools and to output in desired formats/locations/with appropriate attribution. I've already trimmed some fat I think by writing to a scratch folder instead of a scratch personal geodatabase for some interim raster files so they can be stored in the ESRI GRID format vs. the IMG format. I'll check out the profiler suggestions though.

There are some in my office that question Python saying "that compiled code is so much quicker than code running through an interpreter" mainly in comparison to, say, a compiled Visual Basic program or VB.NET program, but that is a good point that the tools are going to take time either way. And, it seems like with present day computing machines that interpreted code may not be that much slower than compiled code to warrant going that extra mile.

EDIT - update on optimization of the program with raster formats.

Wanted to follow up on my "optimization" of this Python program, and I was able to shave 2 hours of processing time by writing interim rasters to GRID format instead of to a personal geodatabase. Not only that, there was a SIGNIFICANT reduction in data size disk space consumption. The original run I did writing all rasters (and they were only point features converted to rasters, and then watershed rasters) resulted in 37.1 GB of data just for those files. Writing the latter two data outputs to a folder in GRID format was reduced to 667 MB of data.

I'd be curious to see how a file GDB would handle these data though mainly in way of the size of the data. But, cutting my processing time down from 9.5 hours to 7.5 hours certainly is enough to advocate for dealing with rasters outside of geodatabases in the GRID format.

Answer 1

First question: how much of this are you doing in Python? Are you just calling out to Geoprocessing tools or are you doing a significant amount of numeric analysis in Python? If the former, the bottlenecks likely live in the tools and using native code in your script won't buy you as much as some other clever workarounds. If the latter, then you may want to find what's slow and make it faster with better algorithms, or possibly numpy, or some other option as discussed below.

py2exe does not actually compile your code to native x86/x64, it just provides an executable that embeds your script as bytecode and provides a mostly portable way of distributing it to users without Python on their systems. It failed when attempting to bundle arcgisscripting, which is why it did not work. Actually getting py2exe working still won't do anything performance-wise.

I very strongly recommend you first use a profiler to identify the slow bits and optimize from there. There is a very good set built in to Python, use cProfile on a long run to find potential places to make it faster. From there you can optimize away sections into custom C or possibly experiment with small portions as Cython .pyx modules.

You can look into Cython for possibly building the whole Python script as a native code extension module, but Psyco may also give you an performance boost with a lower barrier to entry.

Answer 2

How long does the watershed delineation take if run from the standard tools in ArcToolbox as compared to the script version? If the times are similar, then I suspect that there will be no improvement. You might want to consider running long processes in the background outside of ArcMap.

Answer 3

Don't use a personal geodatabase without good reason. In our experience they are consistently much slower than all other forms of esri data storage (ref). Though I have read one report here on GIS.se that saw faster personal than file gdb.

When the workflow consists of many small iterations the call to create the geoprocessor and check out a license is often the most time expensive part of using python. So doing as much as you can either in front of or behind gp = ... (or import arcpy in v10) is one technique I use a lot.

With regard to compiling, this quote says it best:

It's worth noting that while running a compiled [python] script has a faster startup time (as it doesn't need to be compiled), it doesn't run any faster.

Mark Cederholm has a presentation about using ArcObjects in Python with some statistics on shapecopy operations (slide #4). Python doesn't fair very well, running at 32% of what can be achieved with C++ (VBA was 92%, VB & C# at 48%). Don't go running and screaming too quickly, many of the geoprocessing tools are python scripts anyway (search c:\program files\arcgis\ for '*.py').

As many have said in other venues, with python the time spent trying to optimize performance by compiling or writing a C or C++ core function often dwarfs any actual performance gains (possibly) made at runtime. Many say Python's chief benefit is optimizing and improving developer time; human attention is vastly more valuable and expensive than machine processing time.

Answer 4

You cannot compile python code to machine code. When it runs the first time, it is compile to 'bytecode', an intermediate language (which creates pyc files)

py2exe wraps the dll files required by the interpreter and any required python files/external files into an executable. It is not compiled - runtime shouldn't be much different.

It is possible to make Python code run very fast, using a combination of different techniques.

The first thing you should do is profile your code to find the bottlenecks. Once found, I usually use this process:

• Eliminate 'for' loops by using numpy arrays or the map() function. This basically pushes the loop into C.
• Investigate better implementations of the algorithm (this kind of goes concurrently with the above). Stuff like reducing the number of I/O operations, ensuring data is accessed/stored in contiguous blocks.
• Interpreter 'tricks' such as avoiding expensive lookups within loops, avoiding 'if' blocks within loops (use 'try' instead)
• Profile it again
• If it is still too slow, look at pushing critical parts into C using Cython (or writing directly in C, creating a dll and using ctypes to call it)
• Profile again
• If still too slow, look at parallel or GPU computing (multiprocessing library, pyCUDA, ParallelPython etc)

Answer 5

If you import a python script from another location it generates a .pyc file. So, one easy way of testing whether compiling makes a difference would be to turn your script into a function (e.g. main()). If you save that script as example.py then create another file with the following lines:

import example
example.main() # call your script(s)

If you time running from within the script, and running when it is imported, perhaps you can see what the difference is. This is a low-tech way of doing it though.