# Calculating miles between two points using Vincenty formula

I have the following MySQL function:

``````DELIMITER \$\$
DROP FUNCTION IF EXISTS `fn_vincenty_distance`\$\$
CREATE FUNCTION `fn_vincenty_distance` (
lat1 FLOAT, lon1 FLOAT,
lat2 FLOAT, lon2 FLOAT
) RETURNS FLOAT
NO SQL
DETERMINISTIC
COMMENT 'Returns the distance in degrees on the
Earth between two known points
of latitude and longitude
using the Vincenty formula
from http://en.wikipedia.org/wiki/Great-circle_distance'
BEGIN
RETURN  DEGREES(
ATAN2(
SQRT(
END\$\$
DELIMITER ;
``````

I cannot reproduce some of the airport distances in miles. I'm using `69.2 miles` as the distance between each degree of latitude at the equator:

Case 1

SFO-DFW

`fn_vincenty_distance(37.6188056, -122.3754167, 32.896800990, -97.038002010)*69.2`

Returns `1463.70` which is close to `1464.373` from https://www.airmilescalculator.com/distance/sfo-to-dfw/

Case 2

MIA-ORD

`fn_vincenty_distance(25.7953611, -80.2901158, 41.9745219, -87.9065972)*69.2`

Returns `1200.77` which is not close to `1197.062` from https://www.airmilescalculator.com/distance/mia-to-ord/

Question

How can I get the same answer being produced from airmilescalculator.com?

This is to match the calculation by airlines, for example from AA.com:

• you need to work out your 69.2 miles for the correct latitude see gis.stackexchange.com/questions/142326/…
– Mapperz
Commented Jan 10, 2020 at 18:04
• Do "qualifying" miles have any fixed relationship to actual miles? Commented Jan 10, 2020 at 18:19
• I agree with Ian's statement. Your math aside, you're taking the 1197 miles from ORD to MIA as an authoritative fact from this random website. Without knowing how they calculated this value, I wouldn't compare it to real world math. Commented Jan 10, 2020 at 19:50
• You are using the spherical case of the Vicenty's formula , but the site seems to use the ellipsoidal method. Also, just multiply the `ATAN2` output times the radius of the sphere in miles. Commented Jan 10, 2020 at 20:26
• @Mapperz the referenced question calculates at a single point -- how would I do this for two points? Commented Jan 16, 2020 at 2:16