My scale is set to 2. Yet arc (using 10.2) is showing (in my select by attribute - get unique values) my number of 26.67 as 26.670000000000002. My value of 25.25 is shown correctly as 25.25 in the same field.

Again my value of 32.60 is shown as 32.6000000000001.

I have tried replacing the values with the field calculator, but it goes back to these long decimal numbers all on its own.

Why is it doing this and how do I fix it?

These values are not more than 2 decimals, showing more is wrong. but it has to be a double because there are some negatives ..

  • Generating unique values on IEEE floating point values will frequently cause this sort of issue -- Would you prefer a value which will match an equivalence test or one that that uses your preferred formatting? You can't have both.
    – Vince
    May 28 '14 at 19:38
  • I don't completely understand your answer. Preferred formatting is what I want! That's why I set the scale to 2. Anything beyond that is wrong.
    – user30746
    May 28 '14 at 20:08
  • What's your field type?
    – GISHuman
    May 28 '14 at 20:18
  • it's a double, mentioned it, (not so clearly at the end)
    – user30746
    May 28 '14 at 20:37
  • 3
    Some user interface controls may not respect the precision setting when rendering as a string. Remember that in hardware, it's not stored as rounded to anything at all. You should read What Every Computer Scientist Should Know About Floating-Point Arithmetic. May 28 '14 at 20:50

This is more an overall question of how computers deal with and represent numbers and is an important question to grasp for computer programming/GIS in general. Considering the vast amount of papers on this topic I think it is best to give you a few links and they will lead into more research if you are interested on the topic.

One of the classics on the topic is 'What Every Computer Scientist Should Know About Floating-Point Arithmetic' or just go to any number of questions on stack overflow that deal with this topic (look on the right side for links to other similar questions on the topic). One more interesting link that documents how often this question is asked and provides many helpful links (at the bottom of the page) for better understanding.


I'm assuming your field type is float. If you've already set your scale and the numbers are showing up with more decimal points than you specified try this: Right click on the field in question via the attribute table --> properties select numeric From there change the number of decimal places or significant digits. Alternatively, set up a new field and also set your precision to the number of digits you want stored in the field.

Here is a resource outlining the difference between precision and scale

  • no it's a double, have some negatives.. I already set the correct scale and precision. the numeric is set correctly as well. I am not sure you understand or I did not explain myself clearly in the original post.
    – user30746
    May 28 '14 at 20:35

See this link for this answer:

Add Field Discussion at ESRI

When you create float and double fields and specify a precision and scale, if your precision is greater than 6, use a double; otherwise, use a float. If you create a double field and specify a precision of 6 or less, a float field is created. If you create a float field and specify a precision greater than 6, a double field is created.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.