2

I would like to convert double to string using the calculator (or ArcObjects) and get the actual number. Calculator seems to truncate the number at ~15 decimal places but the actual value may be a lot more precise? The only way to i can see the actual value is when i hit the "Get Unique Values" button in the Select by Attributes dialog.

Using a random number obtained from the list of unique Values I can write a simple query that correctly retrieves 13 records:

Shape_Length = 9.999983012676239e-005

The value that shows in the Shape_Length attribute is 0.000100000

if I write the following query by simpley copying this value, it does not retreive any records:

Shape_Length = 0.000100000

Taking this a step further, if i create a column and convert the double values from the Shape_Length field into it, though the numbers apear more precise, they are not identical to the value obtained from the Values list.

When I convert 0.000100000 to a string field using the calculator i get 9.99998301267624E-05

When I query Shape_Length = 9.99998301267624E-05 i do get 13 records in this situation but in other cases i may get less records.

in summary:

Shape_Length value: 0.000100000

Converted to string from Calculator: 9.99998301267624E-05

From Get Unique Values list: 9.999983012676239e-005

that's 3 different numbers. I would like to be basically able to see the same number I see in the Unique Values list in my attribute table. How?

enter image description here

Improve this question
2
  • This may be better asked at Stack Overflow as a pure Python or VBS question
    – Midavalo ♦
    Sep 20 '16 at 18:50
  • See this post about precision/accuracy.
    – Paul
    Sep 20 '16 at 18:58
2

Double and floating point data type only store an approximation of a value, so it can lead to this kind of unexpected behaviour when you try to query the exact values or extract them, etc. So testing double values is always difficult because of the inherent imprecision of double formats.

If you do not need the full precision in your queries I would suggest extracting for example the first 12-14 significant digits and work with those, or query ranges of values.

The values that show up in the tables and the Select By Attributes are formatted, so they are not always exactly equal to the stored value. ArcMap will never show the "exact value", it will always show a maximum of 15-17 digits, because it knows that double precision cannot be more precise. In fact the exact values can have hundreds of digits in base 10 representation, because they are stored in base 2.

Improve this answer
4
  • Thanks. I tried rounding both pairs and it's not a 100% solution. Over a large subset there are always pairs that did not match rounding to less significant digits yielded better results but with increasing the Precision of the rounding more pairs did not match. I did not however try extracting a subset of digits without rounding. Will try.
    – Jakub Sisak GeoGraphics
    Sep 21 '16 at 12:49
  • I have done further research on the double data type, and found out that you need at least 17 digits to guarantee that the conversion will return the same number, thus making the query work. Unfortunately, I don't think ArcMap will show all those digits, so you will always get some cases where it won't work. Have you tried querying a range of values ?
    – FSimardGIS
    Sep 21 '16 at 14:22
  • Thanks. Queering range - another good suggestion! Do you mean to use BETWEEN in the SQL?
    – Jakub Sisak GeoGraphics
    Sep 22 '16 at 14:35
  • That depends on your underlying database type, for example if it is stored in SQL Server then yes, between should work fine. If it doesn't work with your data, try something like FIELD > 1.09 AND FIELD < 1.11
    – FSimardGIS
    Sep 22 '16 at 16:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.