Efficiently projecting datasets of varying coordinate systems into one coordinate system?

Question

I am working on streamlining data processing at my organization. Currently we are using various large-scale Models that we will gradually bring into Python.

The gist of our data processing is as follows:

1. Collect data from sources (State, County, City, etc.)
2. Project all data to a common coordinate system.
3. Perform all sorts of processes on the data.

I am currently concerned with part 2. The data comes in various coordinate systems. Our previous Models would run the ArcGIS Project tool individually on each dataset that needed it. Each Project tool in the Model would be projecting to the same output coordinate system, but some may require different transformation methods (or none at all).

You can imagine having about 70 different datasets whose transformation methods are each manually assigned. It's ugly in code, worse in ModelBuilder. Most importantly, it is hard-coded to those specific datasets and therefore not reusable by another collection of datasets. Ideally, I want to be able to choose a bunch of datasets, point them to a script tool, and reliably output them all in the same coordinate system.

I thought of importing all of the datasets into an ArcGIS Feature Dataset which will project all of the data automatically, but as blah238 pointed out in my previous question, it could be unreliable since it makes a 'best guess' on the transformation methods.

Are there any better methods, or any known tricks or custom tools? My organization is mostly in the Esri world, but I'm open to open-source solutions as well.

Answer 1

Sounds like any way you slice it, either you specify the transformations individually or you don't. Instead of hard-coding things or using ModelBuilder, create a lookup table or spreadsheet (CSV files work well with both Excel and Python) with the datasets and transformations and loop over it programmatically.

Answer 2

I think that in this kind of situation, a real programming (/scripting) solution really starts to shine. For these kinds of analyses (all my analyses actually :)) I use R. It has support for reprojection of data through the proj4 projection library. I suspect that reading and reprojecting the data could be done in under 20 lines of code. Key packages are rgdal for reading and reprojecting data and sp for spatial objects in R. For more detail I refer to the R Task View for spatial data.