3

I want -122d 54m 8.41s W (DMS) to be converted as decimal (DD). I tried two different available online converters to do the calculation. Surprisingly outputs are different. It looks like they match up to 7th decimal places.

-122.902336120571
-122.90233611

Out of curiosity, I used open-source dms-conversion JavaScript library that handles conversion to test what value would be printed out. It came out as:

-122.90233611111111

Only common ground from this observation I can find is up to 7th decimal places matches. I am new to this subject area. I am not sure which decimal value I have to trust if I want it to be converted back as original DMS value.

I wonder each on-line converter and external JS library I tried out has different logic when it comes to truncate seconds?

If I need to implement conversion programmatically either using an external library or math formula available on-line such as this: Converting latitude and longitude to decimal values, how far I have to trust in decimal places for DD value?

4
  • 2
    The difference in location between -122.90233611 and -122.90233612 is about one millimeter. Does that difference matter to your application?
    – Dan C
    Jul 11, 2019 at 17:05
  • 2
    @DanC is absolutely right, more details in Measuring accuracy of latitude and longitude?
    – Taras
    Jul 11, 2019 at 17:12
  • 1
    @DanC, to be honest, I do not know b/c I am not sure how much deviation it would create by one millimeter range but linked article from Taras makes me feel comfortable with having precision up to 7th decimal places. Ty for your time both of you.
    – DaeYoung
    Jul 11, 2019 at 17:17
  • 1
    obligatory xkcd reference
    – Vince
    Jul 12, 2019 at 1:02

2 Answers 2

3

Obligatory XKCD reference:

enter image description here

More detailed answer

1
  • Looks like it is an upcoming Geomanuscript 🙂
    – Taras
    Jul 12, 2019 at 9:20
0

In javascript you only have the data type float available. There isn't a true decimal representation of all numbers to the right of the decimal point. There are some javascript math libraries that will convert to string and back (to decimal) to get as close to "true" as you require for significant digits. It comes down to doing division with binary numbers. I won't list any libraries here because they change. I hope this will help you find your answer.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.