Industry-based examples of using Python in geoprocessing and arcpy

Question

I was asked recently by some students what types of operations GIS Analyst/Developer are usually trying to automate by using Python with geoprocessing in ArcGIS and ArcPy site package. I guess it might be useful to know when searching for some exercises to do to be sure that what you create is relevant for the industry and might be re-used later already at the workplace.

The easiest answer would be "read the Esri help and go through the samples", but I was looking for more specific scenarios which are most common to implement. Thus, sharing workflows sush as "we are getting a .zip file with shapefiles, we use Python to unpack it, project them all to the X coordinate system, load into an SDE geodatabase, grant users access to these data" are most welcome. Please feel comfortable to provide a concise description of the workflow, no extreme details are required.

Internet search does not reveal anything on that, so I thought to ask here. The question seems to comply with the FAQ, since it is a question where a specific answer on a certain issue is expected.

Answer 1

For me, as your question suggests, I use Python a lot for automating batch processing in particular but also for creating any repeatable specialist calculations. These days I don't use ArcPy because I can't afford the ESRI licences as a freelance GIS Consultant. I use GDAL/OGR, Shapely, PostGIS, Numpy and SciPy a lot, though everything in my list could be done with ArcPy (and some of it was). Examples include:

1. Deriving Zonal Statistics for the whole of the UK that first requires mosaicing 20km raster tiles of two different data types, performing some "mapematics" on those rasters, merging the equivalent area of 10km vector polygon tiles, calculating the zonal stats of the result of the raster mapematics and joining the table of stats to the original vector data before outputting to shapefiles in a logical directory structure and burning to CD for the client.
2. Performing sequential visibility calculations every 100m along a road or track and then assigning the results of the calculation as M values back in the route data.
3. Automated process to creating 3D landscape models by mosaicing/merging tiles of raster and vector data, clipping to the required area and then converting to a proprietary (non-GIS) 3D format. I use the little Python library I developed for this a lot in my freelance work.
4. One huge project I worked on in a team used ArcPy to create batch processes to convert or derive new data from GIS data into a format with features which could be consumed by a procedural computer-game-asset-generator. The geoprocessing scripts were called by a batch processing 'slave-driver' also written in Python and running through Django.
5. Python is very useful even for little tasks, particularly where there is any repetition (e.g. feature by feature processing). ArcGIS' Model Builder is much improved with the flow-controls that came in in version 10 but even so, it still frequently can't provide the necessary control and/or it is quicker and easier just to write the process in ArcPy than try to coerce Model Builder.
6. I have created a tool in Python to perform swept-path analysis (to calculate whether a very long vehicle can follow a given route and where the trailer will be likely to get wedged in tight turns between buildings. This is another specialist tooll in by freelance arsenal.
7. Generating output from Mapnik
8. Before ArcGIS became multithreaded, I used Python to allow me to spawn subprocesses which could sometimes speed up long, slow calculations without the overhead of ArcMap cluttering up the memory.

Python in commercial geoprocessing is great because you have all the speed and brevity of scripting that Python provides and the speed of processing the compiled C-style code provides because, while Python is interpreted it is mostly calling compiled C-style code under the hood. Python provides the glue that can hold a lot of sequential geoprocessing tasks together and the list above is just a tiny snapshot of some of the things I personally use it for. In the 'Good Old Days' we would set up a Watch file and have ArcInfo record our command-line input and then clean up the AML (who remembers Arc Macro Language!) to make a reusable process of geoprocessing calls glued together with AML. It's not so different these days, except we use Python or C# as the glue.

Answer 2

I work for a municipality, and the GIS gang provides support to our various departments (Engineering, Building Inspections, Bylaw, Parks, Fire, etc.).

1. Updating parcel and civic address information. We have rather lengthy scripts that manipulate both spatial and attribute data, which involves using various geoprocessing tools to perform spatial stuff as well as connect to relational databases to grab information that is then joined to our spatial data.
2. Custom tools. We have created custom tools using the new Python Add-in Wizard for some of our Non-GIS staff. Some of our staff need to read spatial data as well as perform some basic manipulations. We have designed toolbars that lets them do whatever they need to do without getting into the guts of the ArcGIS environment.

As others have stated, these are just a few examples.

Answer 3

Where to begin...I am a huge proponent of Python in the enterprise, even though I now work in higher ed/state government. Here are some examples of tasks I have used Python for:

1. Moving data around. Pretty simple tasks like routinely moving data around can be automated very easily with Python, especially with the shutil module out of the box.
2. Exporting featureclasses out of an ArcSDE database and into shapefiles for consumption by other software packages. ArcSDE (or some other relational database) is often the master record in an organization, but not every software package can hook into a database. Many packages still will consume good ole shapefiles, and using arcpy, it's painless to export these out nightly so your users can have up-to-date data.
3. Creating spatial datasets from disparate datasets. Everyone in the enterprise uses (and often misuses) Excel to hold their data. With arcpy (or other Pythonic methods) it's easy to take that tabular data that has a spatial component and quickly create a spatial dataset from it. Same goes for text files. I recently created a ArcToolbox tool for a client that reads text files in a proprietary XYZ format and creates ZM-enabled polylines (can't really share much more than that on that one).
4. Translating GIS data so it can be fed into software that has no idea what "spatial" even is. I'm writing tools right now that take GIS datasets (rasters, vectors) and push the data into a 3D modeling program via its Python API. This 3D package cannot work with spatial data formats at all, but it can work with the text values and attributes behind the spatial data. For this I'm using arcpy pull the information out of the geodatabase and push it either to text files or to a XML configuration file.
5. Fetching data. Got a website with a tables of data that you need? Use beautifulsoup to extract it. Got an FTP site with hundreds or thousands of files you need to fetch? Use urllib2 or ftplib to download them easily.

That's just a few examples. What's great about Python in the enterprise is that even without full admin rights to your machine, which is often the case, you can still accomplish quite a bit. Combine that with the gentle learning curve and readability of Python, and you have a great automation tool for the GIS Tech/Analyst that doesn't have a whole lot of programming experience.