enter image description here

enter image description here

The method is not recommended for distances larger than 4km. I am getting reasonable accuracy in terms of the east and north bearings, but for the up direction I am getting error upwards of 1km for distances of 110km (or a 1 degree difference in longitude). The error seems to occur in the WGS84 to ECEF conversion, as the 1km error first appears there. However, the result of the WGS84 to ECEF conversion has been checked against numerous well known and presumably correct oracles such as the results of software like GeographicLib and the Sysense online conversion program.

For example using the WGS84 coordinates of (0,0,0) as a reference point I get the ECEF x,y,z coordinates of 6378137.00, 0.00, 0.00, this is correct. So now using this as a reference point, I then use WGS84 coordinates (0,1,0) and get ECEF x,y,z coordinates of (6377165.58, 111313.84, 0.00). The difference between the two is (−971.42,111313.84,0.00). This means that moving 1 degree east in the WGS84 coordinate system will give a -971.42 meter drop in x direction of the ECEF coordinate system (which is straight down relative to the initial position 0,0,0). This cannot be possible because the curvature of the earth is about 8 inches per mile, and there are 68 miles in 110 km. So the drop should be 544 inches or about -13.8 meters, not -971 meters.

Either the two oracles that the result has been checked against are incorrect, or the error is an inherent problem with the WGS84 conversion formula.

  • The earth surface has to curve 6378.137 km in 90 degrees. It is lower near the equator where the curve is shallower (it's around 111 km near the pole).
    – mkennedy
    May 27, 2016 at 23:52
  • Hard to say without providing the basis, and showing your working. Perhaps you are understanding or applying the formulas incorrectly. In general, the conversion to the ellipsoid should be close (not kilometres).
    – BradHards
    May 28, 2016 at 5:47
  • Could you tell me where you took this formulae from?
    – tommsch
    Mar 2, 2021 at 8:22

2 Answers 2


The implementation here actually appears correct. The problem exists in what is being checked, and what is expected. The distance checked is the longitude while the expected distance is the latitude.

We can perform two quick checks to satisfy that the our conversions are correct:

  1. The magnitude of the ECR vector should be ~637.814 km at the equator.

For both of the given vectors I found their magnitude to be 637.814 km.

  1. The longitudinal change at the equator should be about 111.321 km.

The straight line distance between the values given is approximately 111.318 km. This is actually only ~3m off from what should be expected. The difference is because I did not take into account the curvature of the earth.

Now as to why is the difference (-0.971 km, 111.313 km)? We must remember that ENU is a tangent plane and X is merely a distance to the plane. Also since 1 km along the plane is not 1 km along the surface of the earth, we cannot expect to see X slope away at the given rate.

If none of that helps, then this answer may help Longitude Length ...in miles


To clarify, WGS84 is an Earth Centered Earth Fixed (ECEF) reference frame and utilizes both ECEF and latitude, longitude, and altitude (height) coordinates. Conversion is required from geocentric (x,y,z) to geodetic (lat, long, alt) coordinates.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.