Three different distance values, which one should I use?

Question

I'm implementing a script that (ideally) has a threshold of <1 meter. However, I have three different distances being pulled from these points that differ by multiple meters. I was wondering which I should go by.

From my understanding, geopy uses a model of some sort for a more accurate number whereas mpu assumes a sphere and uses a "haversine" formula. The other distance derived referred to as "delta" in the script is what was there previously and I'm not sure how to interpret that one in terms of comparing it to the other two. Is there one you would recommend I use? It is reading a random set of survey points, usually around the coast of US.

deg_to_m = 111139  
lon_scale = np.cos(np.deg2rad(new_coords[1]))
delta_x = lon_scale * deg_to_m * (old_coords[0] - new_coords[0])
delta_y = deg_to_m * (old_coords[1] - new_coords[1])
delta_dist = np.hypot(delta_x, delta_y)

print(f"delta distance from point {new_coords} and {old_coords}= {delta_dist}")
print(f"geopy distance from point {new_coords} and {old_coords} = {geopy.distance.geodesic((new_coords[1],new_coords[0]), (old_coords[1],old_coords[0]) ).m}")
print(f"mpu   distance from point {new_coords} and {old_coords} = {(mpu.haversine_distance((new_coords[1],new_coords[0]), (old_coords[1],old_coords[0])) )*1000}")

Sample Results

delta distance from point (-73.54103769993867, 40.967273802785186, 17.68) and (-73.55888019993945, 40.96432750278418, 18.35)= 1532.7166288425676
geopy distance from point (-73.54103769993867, 40.967273802785186, 17.68) and (-73.55888019993945, 40.96432750278418, 18.35) = 1537.1838851634725
mpu   distance from point (-73.54103769993867, 40.967273802785186, 17.68) and (-73.55888019993945, 40.96432750278418, 18.35) = 1533.5205811777234

delta distance from point (-73.54103769993867, 40.967273802785186, 17.68) and (-73.55625289993931, 40.96710510278517, 18.32)= 1276.9866000176955
geopy distance from point (-73.54103769993867, 40.967273802785186, 17.68) and (-73.55625289993931, 40.96710510278517, 18.32) = 1280.9052835621858
mpu   distance from point (-73.54103769993867, 40.967273802785186, 17.68) and (-73.55625289993931, 40.96710510278517, 18.32) = 1277.6308281008874

Answer 1

Your "delta" distance calculation is essentially using the Pythagorean theorem to compute distances. This works on a plane, but the distances you want to calculate are on the curved Earth's surface, hence this is only a quick approximation that would only work for very small distances away from the poles, and would tend to approach the haversine distance if the reference Earth radius is the same. For longer distances, the result can be wildly off reality because it is assuming a planar representation of latitudes and longitudes.

The haversine distance is the great-circle distance, and can be suitable for some applications, it gives the correct distance within about 0.5%.

Your geopy distance uses a geodesic distance, assuming the Earth is an spheroid instead of a sphere, hence it is more accurate for distance calculation on the Earth.

So I would recommend sticking with the geodesic distance, especially if your distances can be longer. We can see that even at 1500m, the difference between haversine and geodesic is already over 3 meters in your example.

Note that none of these formulas take into account elevations. The distance is computed on the reference sphere or spheroid surface.