Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

My scale is set to 2. Yet arc (using 10.2) is showing (in my select by attribute - get unique values) my number of 26.67 as 26.670000000000002. My value of 25.25 is shown correctly as 25.25 in the same field.

Again my value of 32.60 is shown as 32.6000000000001.

I have tried replacing the values with the field calculator, but it goes back to these long decimal numbers all on its own.

Why is it doing this and how do I fix it?

These values are not more than 2 decimals, showing more is wrong. but it has to be a double because there are some negatives ..

share|improve this question
    
Generating unique values on IEEE floating point values will frequently cause this sort of issue -- Would you prefer a value which will match an equivalence test or one that that uses your preferred formatting? You can't have both. –  Vince May 28 at 19:38
    
I don't completely understand your answer. Preferred formatting is what I want! That's why I set the scale to 2. Anything beyond that is wrong. –  user30746 May 28 at 20:08
    
What's your field type? –  GISKid May 28 at 20:18
    
it's a double, mentioned it, (not so clearly at the end) –  user30746 May 28 at 20:37
3  
Some user interface controls may not respect the precision setting when rendering as a string. Remember that in hardware, it's not stored as rounded to anything at all. You should read What Every Computer Scientist Should Know About Floating-Point Arithmetic. –  Jason Scheirer May 28 at 20:50

3 Answers 3

This is more an overall question of how computers deal with and represent numbers and is an important question to grasp for computer programming/GIS in general. Considering the vast amount of papers on this topic I think it is best to give you a few links and they will lead into more research if you are interested on the topic.

One of the classics on the topic is 'What Every Computer Scientist Should Know About Floating-Point Arithmetic' or just go to any number of questions on stack overflow that deal with this topic (look on the right side for links to other similar questions on the topic). One more interesting link that documents how often this question is asked and provides many helpful links (at the bottom of the page) for better understanding.

share|improve this answer

I'm assuming your field type is float. If you've already set your scale and the numbers are showing up with more decimal points than you specified try this: Right click on the field in question via the attribute table --> properties select numeric From there change the number of decimal places or significant digits. Alternatively, set up a new field and also set your precision to the number of digits you want stored in the field.

Here is a resource outlining the difference between precision and scale

share|improve this answer
    
no it's a double, have some negatives.. I already set the correct scale and precision. the numeric is set correctly as well. I am not sure you understand or I did not explain myself clearly in the original post. –  user30746 May 28 at 20:35
    
+1 for right clicking and changing the properties –  Sara Barnes Jun 4 at 14:57

See this link for this answer:

Add Field Discussion at ESRI

When you create float and double fields and specify a precision and scale, if your precision is greater than 6, use a double; otherwise, use a float. If you create a double field and specify a precision of 6 or less, a float field is created. If you create a float field and specify a precision greater than 6, a double field is created.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.