3

I'm using ModelBuilder and have three variables (var1, var2 and var3) which I want to use to calculate a new value. This value should be used as an input for the "Minus" tool. The calculation to be performed looks like this:

NewValue = var1 + (var2 / 2 / var3)

When calculating this manually for var1 = 2, var2 = 2 and var3 = 3 I get 2.333... (2 + (2 / 2 / 3))

When using the "Calculate Value" tool I get: 2

enter image description here

I was forced to use the data type "Formulated Raster" as output for "Calculate Value" because it seems to be impossible to use the output as an input for the "Minus" tool otherwise. Maybe this is the reason for the rounding.

So I tried using the "Raster Calculator" instead but this just leads me to an error message I don't understand:

enter image description here

How can I get my Model to calculate and use the correct value of 2.333...?

EDIT:

Changing the Rastercalculator Expression like suggested in the comments doesn't fix the problem:

enter image description here

  • 1
    Try float(var1 + (var2 / 2 / var3)) – cartoscience Jul 31 '16 at 14:56
  • When you populate the Calculate Value tool what are you setting the parameter Data Type to? In your case you must set it to Double as well as following @Ali's advice. – Hornbydd Aug 5 '16 at 11:13
  • @ Hornbydd This was my first guess but as I wrote above "I was forced to use the data type "Formulated Raster" as output for "Calculate Value" because it seems to be impossible to use the output as an input for the "Minus" tool otherwise. " – Simon Aug 5 '16 at 11:19
2

For python division to result in a float data type you must either have the numerator or demoninator as a float. Consider 1 divided by 2:

>>> 1/2
0

The answer is 0.5 but is output as the integer component 0. In your case the answer 2 results from 2 + (0) - even though we know it should be 2 + (0.3333).

Now, back to 1 divided by 2. Making either number a float, explicitly or implicitly, will give the result as a float:

>>> 1/float(2) # explicit
0.5

>>>float(1)/2 # explicit
0.5

>>>1.0/2 # implicit
0.5

Therefore, changing your expression to something along the lines of:

2 + (2/2/float(3))

should solve your problem getting your intended value.

Additionally, setting the output to Double in the Calculate Value tool should enable it to be used in the Minus tool.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.