1

I have the following MySQL function:

DELIMITER $$
DROP FUNCTION IF EXISTS `fn_vincenty_distance`$$
CREATE FUNCTION `fn_vincenty_distance` (
        lat1 FLOAT, lon1 FLOAT,
        lat2 FLOAT, lon2 FLOAT
     ) RETURNS FLOAT
    NO SQL
    DETERMINISTIC
    COMMENT 'Returns the distance in degrees on the
             Earth between two known points
             of latitude and longitude
             using the Vincenty formula
             from http://en.wikipedia.org/wiki/Great-circle_distance'
BEGIN
    RETURN  DEGREES(
    ATAN2(
      SQRT(
        POW(COS(RADIANS(lat2))*SIN(RADIANS(lon2-lon1)),2) +
        POW(COS(RADIANS(lat1))*SIN(RADIANS(lat2)) -
             (SIN(RADIANS(lat1))*COS(RADIANS(lat2)) *
              COS(RADIANS(lon2-lon1))) ,2)),
      SIN(RADIANS(lat1))*SIN(RADIANS(lat2)) +
      COS(RADIANS(lat1))*COS(RADIANS(lat2))*COS(RADIANS(lon2-lon1))));
END$$
DELIMITER ;

I cannot reproduce some of the airport distances in miles. I'm using 69.2 miles as the distance between each degree of latitude at the equator:

Case 1

SFO-DFW

fn_vincenty_distance(37.6188056, -122.3754167, 32.896800990, -97.038002010)*69.2

Returns 1463.70 which is close to 1464.373 from https://www.airmilescalculator.com/distance/sfo-to-dfw/

Case 2

MIA-ORD

fn_vincenty_distance(25.7953611, -80.2901158, 41.9745219, -87.9065972)*69.2

Returns 1200.77 which is not close to 1197.062 from https://www.airmilescalculator.com/distance/mia-to-ord/

Question

How can I get the same answer being produced from airmilescalculator.com?

This is to match the calculation by airlines, for example from AA.com:

enter image description here

7
  • you need to work out your 69.2 miles for the correct latitude see gis.stackexchange.com/questions/142326/…
    – Mapperz
    Jan 10, 2020 at 18:04
  • Do "qualifying" miles have any fixed relationship to actual miles?
    – Ian Turton
    Jan 10, 2020 at 18:19
  • I agree with Ian's statement. Your math aside, you're taking the 1197 miles from ORD to MIA as an authoritative fact from this random website. Without knowing how they calculated this value, I wouldn't compare it to real world math.
    – KHibma
    Jan 10, 2020 at 19:50
  • You are using the spherical case of the Vicenty's formula , but the site seems to use the ellipsoidal method. Also, just multiply the ATAN2 output times the radius of the sphere in miles. Jan 10, 2020 at 20:26
  • @Mapperz the referenced question calculates at a single point -- how would I do this for two points?
    – Kermit
    Jan 16, 2020 at 2:16

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.