2

In model build, I have the following Field Calculator tool running:

enter image description here

You can see in the results below that when the calculation involves a decimal, the output shows 6 decimal points, but when the input field is an integer, the result is rounded to an integer. Both input fields and the output fields are data type "double."

enter image description here

What am I doing wrong here? My desired result is to output the decimal, not the integer. (This is percent change, so there isn't much point in just rounding to 0, 1, or -1.)

3

Try return (new -old)/float (old)

This document explains why.

quoted:

...division poses a problem: if the expressions for both arguments happen to have an integral type, it implements floor division rather than true division. The problem is unique to dynamically typed languages: in a statically typed language like C, the inputs, typically function arguments, would be declared as double or float, and when a call passes an integer argument, it is converted to double or float a the time of the call. Python doesn't have argument type declarations, so integer arguments can easily find their way into an expression.

The correct work-around is subtle: casting an argument to float() is wrong if it could be a complex number; adding 0.0 to an argument doesn't preserve the sign of the argument if it was minus zero. The only solution without either downside is multiplying an argument (typically the first) by 1.0. This leaves the value and sign unchanged for float and complex, and turns int and long into a float with the corresponding value.

  • 1
    Thanks. But why does this happen and why does this solution work? – matthewiannowlin Mar 3 '16 at 20:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.