Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

Version control is an indispensable tool for software development, allowing one to reliably and cleanly step back in time to the last time X did it's job exactly right, or to see what changed between then and now -- typically used when trying to figure out why X is no longer working exactly right.

However all the tools I know about for this work only on plain text files. Toolboxes (standard ones, not the python toolboxes introduced in 10.1), and thus their models, are binary. Does anyone have a workable method to bring versioning to them?

Note: versioning is different from backup. There are any number of simple methods to create snapshots of files for a particular date/time -- Windows backup, previous versions, xcopy /s d:\foobar\ x:\foobar_%date%, zip stuff_%date%.zip stuff\*, and so on.

Applying a tool such as git, fossil, mercurial, subversion, or ... to a binary file is one step better than using xcopy or zip in so much as one can add a commit message, "Model foobar %date% now overwrites previous results only if Baz doesn't exist", but is still anemic compared to what that same toolset can do applied to text files: e.g. show me exactly what was changed between last year and today.

WinMerge screenshot

share|improve this question
1  
If (and it's a big IF with ArcGIS) you're using PostGIS, PostGIS versioning has been released - kappasys.ch/cms/index.php?id=23&L=5 (credit to slashgeo.org for the link) –  om_henners Dec 1 '10 at 3:32
2  
@om_henners this Q is about versioning geoprocessing models not spatial data. –  matt wilkie Nov 28 at 16:58

4 Answers 4

Every mainstream version control software, be it central central version control like SVN or distributed solutions like Git, Mercurial, Bazaar etc. allow storage of binary files. They are all quite effective both performance-wise and also in terms of occupied space.

Inspecting differences between revisions/versions of a file is of course different story. There is not much you can do when you want to compare ArcGIS models, which indeed are binary. I do not know of any diff tools for models, I doubt there will be any as most people do not need this specific functionality.

share|improve this answer
    
But you would still be able to roll back and/or retrieve previous versions, right? –  Chad Cooper Nov 22 '10 at 21:13
1  
Well yes, but I think the question is more along the lines of seeing the actual differences between models, if I understand it correctly. –  Petr Krebs Nov 22 '10 at 21:20
    
That's right. I'm already putting the binary .tbx file in VC (using mercurial) but that's not very much different from just retrieving the same file from regular backup. –  matt wilkie Nov 23 '10 at 18:01
    
git attributes allows you to use simple programs to version some binary files--there's already programs for .docx and image exif metadata. ArcGIS models would need a similar custom program. –  James Conkling Nov 30 at 18:55
up vote 4 down vote accepted

Currently I have the workflow of ArcCatalog: opening toolbox > selecting model > editing > file > export > to python, switch to SCM tool > refresh changes > commit changes (enter log comment).

It's cumbersome so I don't do it so much, and thereby lose many of the benefits of versioning.

share|improve this answer
1  
accepting the answer for now, because it's what I'm doing. I'll happily switch to a better one if it shows up! –  matt wilkie Nov 25 at 23:19
    
no problem matt. anyway you cant win your own bounty –  Below the Radar Nov 28 at 17:37
    
@below i'm aware of that, after solutions not bounty (and accepted answer before i opened the bounty anyway) –  matt wilkie Nov 28 at 19:51

The introduction of Python toolboxes at ArcGIS 10.1 for Desktop invalidates your four year old statement that all:

Toolboxes, and thus their models, are binary.

Standard toolboxes are binary but Python toolboxes (*.pyt) are text files.

Consequently, I think Python toolboxes should be considered if version control of source code trumps the requirement for a model building GUI.

You are aware of this from your answer on Why learn/use Python Toolboxes over Python Script Tools? but I thought I should include this as an answer here so that the option to use Python toolboxes (to gain easy access to version control) rather than Standard toolboxes is not overlooked by future readers of this Q&A.

share|improve this answer
    
thanks for drawing out this important distinction. It's unfortunate that the same word, toolbox, is used for what are in fact quite different creatures. I'll touch up the wording accordingly. –  matt wilkie Nov 25 at 23:08

ModelBuilder is old, clunky, and is not getting any significant updates with ArcGIS Pro, if this tweet is any indication. I have never been a big fan of it (though begrudgingly still use it when I have to), so you might consider this answer as a sidestepping of the question and a recommendation to look at alternatives.

FME is arguably the most obvious ModelBuilder alternative, as it has a similar flow diagram UI. One relevant advantage is that its documents are in plain text format, so they can diffed (although there is often a lot of meaningless, auto-generated cruft that you have to learn to ignore). It is commercial software, however, so its cost may be out of reach for some.

Others which I'm less familiar with include Orfeo Toolbox, Whitebox Geospatial Analysis Tools, and QGIS's graphical modeler (based on SEXTANTE). These are all open source modeling environments with GUIs.

A big push that I have observed at GIS and open data conferences in recent years is towards the idea of "reproducible research", that is, data and processes that can be easily shared with and reproduced by others. That often means using open data formats, open source software, and shared repositories. Python and R are very popular for this.

I thought Dharhas Pothina's presentation on Python and GIS earlier this year made a good argument for this. I agree quite strongly that over-reliance on a GUI is detrimental to reproducibility. With code, so long as you are familiar with the language, you can scan through it pretty quickly and figure out what's going on, whereas with a GUI you have to click and scroll around through lots of different windows, often nested deep within one another, to get at values and settings.

Certainly, there is a tradeoff here, but in my opinion, anyone doing any serious work (scientific research, policy-making, etc.) should be using tools which facilitate reproducibility.

Sorry that this doesn't answer your question directly (though I don't believe there is an easy answer).

share|improve this answer
2  
Could not agree more about the concept of reproducible research. To me it is the most compelling reason why researchers should be using OSS. –  djq Dec 1 at 23:12
1  
I couldn't agree more about heavy reliance on GUI often being detrimental to reproducibility. The rub with code is in "so long as you are familiar with the language". The small size of that gate keeps a lot of smart and productive people outside, in the wilderness. Really the core of this question is searching for a way to widen that gate. It's frustrating because Modeller almost has both GUI and code. You're right, it is withering on the vine from lack of attention. Sad, we have people who are just now finding mojo in automation and scripting, through the approachability of Modeller. –  matt wilkie Dec 5 at 18:47

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.