3

I want -122d 54m 8.41s W (DMS) to be converted as decimal (DD). I tried two different available online converters to do the calculation. Surprisingly outputs are different. It looks like they match up to 7th decimal places.

-122.902336120571
-122.90233611

Out of curiosity, I used open-source dms-conversion JavaScript library that handles conversion to test what value would be printed out. It came out as:

-122.90233611111111

Only common ground from this observation I can find is up to 7th decimal places matches. I am new to this subject area. I am not sure which decimal value I have to trust if I want it to be converted back as original DMS value.

I wonder each on-line converter and external JS library I tried out has different logic when it comes to truncate seconds?

If I need to implement conversion programmatically either using an external library or math formula available on-line such as this: Converting latitude and longitude to decimal values, how far I have to trust in decimal places for DD value?

  • 2
    The difference in location between -122.90233611 and -122.90233612 is about one millimeter. Does that difference matter to your application? – Dan C Jul 11 at 17:05
  • 2
    @DanC is absolutely right, more details in Measuring accuracy of latitude and longitude? – Taras Jul 11 at 17:12
  • 1
    @DanC, to be honest, I do not know b/c I am not sure how much deviation it would create by one millimeter range but linked article from Taras makes me feel comfortable with having precision up to 7th decimal places. Ty for your time both of you. – DaeYoung Jul 11 at 17:17
  • 1
    obligatory xkcd reference – Vince Jul 12 at 1:02
2

Obligatory XKCD reference:

enter image description here

More detailed answer

  • Looks like it is an upcoming Geomanuscript 🙂 – Taras Jul 12 at 9:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.