Modern mathematics characterizes transformations in terms of the geometric properties that are preserved when the transformations are applied to features.
A time-honored example is the set of Euclidean transformations of the plane: these are the ones that preserve all distances and (unoriented) angles. The study of this group of transformations is the subject of Euclidean geometry and the source of all those congruence theorems about triangles we all learn in school. It turns out that these transformations have relatively simple and illuminating descriptions. For instance, they consist of all translations, rotations, improper rotations (a rotation followed by a reflection) and skew-translations (a translation followed by a reflection). They can also be characterized as all transformations achievable by applying a set of reflections. The point is that these descriptions of the Euclidean group are derived from its characterization as distance-preserving rather than the other way around.
For mapping, analysis, and georeferencing purposes we usually value one particular set of properties over all others: area when computing areas, orientation when computing directions, distance when computing distances, (local) angles when computing angles, similarity when comparing shapes, incidence and inside versus outside when performing topological comparisons, and so on. In each case there is a group of invertible transformations of the plane that preserves the desired properties.
There's nothing deep about this: if you specify any property of features in the plane, then when the transformation F preserves them and the transformation G preserves them, the transformation GF (F followed by G) must preserve them, too. There is always a transformation that will preserve any property: the identity transformation (everything stays put). So you form the collection of all such transformations. If you insist that they be invertible--that is, they can be undone--then this collection forms a mathematical object called a "group." The deep part is that any group of transformations can be conceived of as defining a "property" of features in the plane. A modern mathematician would say, for instance, that "shape" is whatever is preserved by the group of Euclidean transformations and isotheties (see below). That's the definition of "shape"!
Although the foregoing answers the question--you choose among transformations that preserve properties which matter for your analysis--it ducks the more practical issue of how to find these groups of transformations and how to compute with them. Here is a partial guide to this menagerie organized by the property they are intended to preserve. I describe each group, how to compute with it, and how it is usually used in practice.
Size and orientation. This is the group of translations. A translation is given by a vector (e, f); it acts on an arbitrary Cartesian point (x, y) by sending it to (x + e, y + f). The translations shift features around without rotating them or changing their sizes. When coupled with isotheties (see below), this is the group that pans and zooms maps. It often is used to provide "world coordinates" for images (where rotation is not needed).
Size (distance). This is the group of Euclidean transformations, or isometries. It includes the translations as well as all reflections and rotations. A rotation through angle q sends (x, y) to (x * cos(q) - y * sin(q), x * sin(q) + y * cos(q)). This is the group to use when comparing shapes and studying the usual Euclidean properties of shape, including length and curvature.
Shape. In addition to the Euclidean transformations, this group includes all isotheties: these amount to a uniform rescaling with respect to a central point. For instance, an isothety with respect to the origin (0, 0) sends (x, y) to (a*x, a*y) for some non-zero number a. This group is used when lengths and areas only need to be compared rather than measured on an absolute scale.
Incidence. These are the projective transformations. They correspond to what a camera records when it views features on a planar surface in three dimensions: all lines are rendered as lines, but they can be foreshortened and rotated. All the foregoing are projective transformations, but there exist some additional ones. The most general is given by a collection of numbers (a, b, c, d, e, f, g, h) and sends (x, y) to the value (u, v) with u = (a*x + b*y + c)/(g*x + h*y + 1) and v = (d*x + e*y + f)/(g*x + h*y + 1). There is a mild restriction on the possible values of (a, b, ..., h). Projective transformations evidently are useful for re-rendering maps made from cameras (such as aerial or satellite photos) to present them from alternative points of view. The most general "world file" formats used to provide coordinates for images or raster datasets specify projective transformations and usually use the numbers (a, b, ..., h) to do so.
Area. Some projective transformations preserve areas: the special affine transformations. These are the ones that can be written in terms of four numbers a, b, c, d for which a*d - b*c is either +1 or -1. They send (x, y) to (a*x + b*y), (c*x + d*y). These are useful for converting among images of area-preserving projections. For instance, any two cylindrical equal-area projections of an ellipsoid differ by a special affine transformation.
Angle. These are the conformal transformations. Unlike the preceding ones, they cannot be parameterized with a finite set of real numbers. Conformal transformations are best understood as transformations of the complex numbers, the analytic ones. They are precisely the ones that have a complex derivative. All have formulas in terms of power series and integrals. They often are obtained as solutions of differential equations. Because preserving angles tends to render shapes fairly accurately (at least small shapes), conformal transformations are sometimes referred to (incorrectly) as "shape-preserving." There are lots more of them than there are Euclidean transformations, though, so they provide more flexibility in transforming maps than just rotations and translations. The (small) price paid is that larger shapes are distorted.
Circular arcs (including arcs of "infinite radius" circles--straight lines). These are the Mobius transformations. In terms of the complex number z = x + y*i (with i^2 = -1), they can all be written in terms of four complex numbers a, b, c, and d with a*d - b*c = 1; z is sent to the complex number (a*z + b)/(c*z + d). Mobius transformations are useful when working with configurations of circular and linear features. (For instance, images of lines of latitude and longitude in a Stereographic projection of the sphere are all portions of circles and lines in the plane.) Being complex analytic (except where c*z + d = 0), they are conformal almost everywhere.
Incidence, nearness, and inside/outside. These are the continuous transformations. Many cannot even be written in terms of formulas: their defining quality is that around any point (x, y) there will always be infinitely many points (x', y') close to it which do not become greatly separated when they are transformed. Continuous (invertible) transformations provide an enormously rich and flexible way to turn almost any map into anything else while still maintaining important relationships among features, such as incidence, intersection, disjointness, and containment. This provides opportunities to simplify such analyses and make them more efficient.
This list is by no means exhaustive, but it does cover most of the groups of transformations commonly used in two dimensions.
Polynomial transformations do not fit in this list for several reasons. First, most polynomial functions are not invertible, so polynomials tend to be useful only within bounded regions of the plane. Second, polynomials do not preserve any practically useful property. (There is a deeper sense in which they preserve something, but that is a formal algebraic structure which rarely has a direct application in GIS or geography.) Polynomial functions are used solely because they are relatively simple to specify and to calculate, since they can be computed using a small number of multiplications and additions. There are enough of them that they can be used as a relatively flexible way to approximate any continuous function (q.v.), thereby overcoming the difficulty that exact formulas do not exist for most continuous functions.
Typically, polynomial transformations are not used by themselves, but rather as building blocks of various kinds of splines. A spline is locally polynomial, but its coefficients are allowed to change in a controlled fashion throughout the plane. Splines are used in "warping" one image to correspond to another, for instance. In this fashion any map can be made to correspond as closely as you would like to any other map of the same features, no matter how arbitrary the distortion.
