I'm fairly new at calculating distances with precision, but I wanted to know what is the best formula for calculating short distances in UTM coordinates.

The distances that I'am calculating are all less than 30 km and all coordinates lie at Central America.

For instance, the pythagorean distance could cause too much bias?

If not, would it still be a respectable choice either way?

If possible, base your answer on some sort manual or academic paper (and make a reference to it).

  • 1
    The difference is likely to be insignificantly small. Pythagorean/Cartesian distances work for me. The only other realistic choice would be geodetic based on the Earth Geodetic Model but that's a lot of maths for only a very small refinement. Both methods are 'as the crow flies' and don't take topography into account - a surface distance is a completely different thing. – Michael Stimson Nov 3 '14 at 1:43
  • But, Michael, could you give me a reference for your opinion? I am writing a document in which I feel need to justify why I am using the pythagorean distance. – John Doe Nov 3 '14 at 1:53
  • 1
    no, only logic. – Michael Stimson Nov 3 '14 at 2:41
  • 1
    I just noticed you said "best" but didn't say how accurate. Do you need +/- 1%, 0.1%, ... 0.0001% accuracy? – Martin F Nov 3 '14 at 7:12
  • 1
    For highest accuracy, you've got two choices: reverse project your UTM coordinates to geographic and do geodetic (spherical trigonometry) distance calculations (along the lines suggested by Mike T); or keep everything in UTM and calculate the scale factors to correct the simple pythagorean distance (see my answer). Neither one is simple unless you have the right tools. – Martin F Nov 3 '14 at 7:17

The best way I can think is to get two UTM points, convert them to Lat/Long, and compare their geodesic distances to their UTM pythagorean distance.

E.g. Take a point from this example:

The CN Tower is ... in UTM zone 17, and the grid position is 630084m east, 4833438m north.

So if we take A (17n 630084 4833438) and move it 30 km east, we get B (17n 660084 4833438). Convert these to Lat/Long (using GeoConvert), where A' (43°38'33.2224"N 079°23'13.7143"W), and B' (43°38'12.1573"N 079°00'55.3849"W). Now, find the inverse geodesic between A' and B' (using GeodSolve), and the resulting length s12 is 30004.205 m, or 0.0140% larger.

Similarly, moving A 30 km north is C (17n 630084 4863438) or C' (43°54'45.2518"N 079°22'47.5212"W), where the distance between A' and C' is 30005.647 m or 0.0188% larger.

We see that the UTM projection around the CN Tower is underestimating the true distance by about 0.016%. Note that there is anisotropy in the warping of the transverse Mercator projection, depending on the azimuth of the two points in the analysis. An analysis in a different part of the world will yield a different answer, so do your own analysis on a bunch of pairs of points from your region of interest.

Literary references are available on the pages I linked to.

  • 2
    GeoConvert (written by me) will compute the UTM scale for you. In the first example, the scale at the mid-point is given by "echo 17n 645084 4833438 | GeoConvert -c -p 3". The second number 0.9998589143 is the scale. 30000/0.9998589143 = 30004.233 m which is within 3cm or 0.0001% of the true answer. – cffk Nov 3 '14 at 17:02
  • That is a spectacular answer Mike! I notice the difference is around 5 metres over 30km. Is the data you're measuring it from accurate to less than 5m? Like I said, the difference between methods is a lot of maths for a refinement less than the general accuracy of GIS data is likely to be... and it's still not an over land (walking) distance. Calculation to this level of refinement would be required for drones, missiles or measuring continental plates over time. – Michael Stimson Nov 3 '14 at 23:38

If you need high accuracy distances, or "ground" distances, you need to convert your UTM "grid" distances (which you do indeed calculate via pythagorous) using a combined scale factor. This removes the distortion introduced by the combination of (a) reducing the horizontal distance at its elevated (above the ellipsoid) position on the earth and (b) projecting this length from the curved ellipsoid onto the flat map.

ground distance = grid distance / combined scale factor
combined scale factor = elevation scale factor * projection scale factor

It is the kind of conversion that land surveyors do often. UTM combined scale factor values typically range between approximately 0.999 and 1.001 but the actual values depend on how high you are above or below the ellipsoid (or datum) and on how far east or west you are from the UTM zone's standard lines.

The UTM projection scale factor can be calculated from a rather complicated formula or from look-up tables.

Note that for long lines (5km or more), use the Simpson rule for an average projection scale factor:

S = (S1 + 4 Sm + S2) / 6

in other words, one sixth of each of the two end factors plus two thirds of the middle factor

Also note that the elevation scale factor is sometimes called a "sea-level" scale factor but it is a function of ellipsoidal elevation, h, which is the sum of height above the geoid (sea-level), H, and the geoid-ellipsoid separation, N:

h = H + N

Your height above the sea-level is found on topographic maps. Your geoid-ellipsoid separation is generally found in geodetic look-up tables -- there is no simple calculation for it.

The elevation scale factor, ESF is then

ESF = (R + h) / R

where R is the Earth's radius.

For references, please search books on surveying or geodesy or search for various of the above key terms.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.