I've been coding with Python for several months now and have developed some reasonably complex scripts for primarily geoprocessing tasks. That being said, I'm still learning a lot as I'm coming from a SQL/VBA/VBScript background.
I know that compiled code typically runs faster than code that must be processed by a language interpreter, so I'm interested in the possibility of compiling a geoprocessing Python script to a .EXE file for working with big data.
Is this even possible? If it is, what i the best way to compile a Python (.py) script that is importing the arcgisscripting or arcpy modules?
I spent a few minutes trying what I found doing some quick Google searches that returned this article among others: http://www.ehow.com/how_2091641_compile-python-code.html
The compiler seemed to work, but upon executing the resulting .EXE file it gave a cryptic error saying some files were unavailable.
The Python script runs what seems to be reasonably well from the command line, but I'm wondering if I could see some slight improvement if I were able to compile the .py file. Again, I'm working with some big datasets that are taking +20 hours to process (delineating watersheds from input water-quality sample sites). I'll take anything I can get in way of improvements.
The script ran 10% quicker outside of ArcGIS from the command line using a test set of sites versus setting the script up as a script tool in a new toolbox in ArcCatalog. I've been running the script from the command line w/o any instance of ArcGIS open on a dedicated machine.
So, is it possible to compile Python scripts that import the arcgisscripting module and that call ArcToolBox tools?
Thanks for the input, this is helpful for me. The script is largely a way to coordinate a number of ArcGIS tools and to output in desired formats/locations/with appropriate attribution. I've already trimmed some fat I think by writing to a scratch folder instead of a scratch personal geodatabase for some interim raster files so they can be stored in the esri GRID format vs. the IMG format. I'll check out the profiler suggestions though
There are some in my office that question Python saying "that compiled code is so much quicker than code running through an interpreter" mainly in comparison to, say, a compiled Visual Basic program or VB.NET program , but that is a good point that the tools are going to take time either way. And, it seems like with present day computing machines that interpreted code may not be that much slower than compiled code to warrant going that extra mile
EDIT - update on optimization of the program with raster formats
Wanted to follow up on my "optimization" of this Python program, and I was able to shave 2 hours of processing time by writing interim rasters to GRID format instead of to a personal geodatabase. Not only that, there was a SIGNIFICANT reduction in data size disk space consumption. The original run I did writing all rasters (and they were only point features converted to rasters, and then watershed rasters) resulted in 37.1 GB of data just for those files. Writing the latter two data outputs to a folder in GRID format was reduced to 667 MB of data.
I'd be curious to see how a file GDB would handle these data though mainly in way of the size of the data. But, cutting my processing time down from 9.5 hours to 7.5 hours certainly is enough to advocate for dealing with rasters outside of geodatabases in the GRID format.