3

I have a polygon featureclass containing an integer field 'gridcode'. I've added 2 new float fields VB_calc and Python_calc, both of which should be equal to gridcode / 1,000,000.

If I calculate the field using the Field Calculator with the VB Script [fieldname] syntax, the result is as expected.

enter image description here

If I use the Python !fieldname! syntax, the results are all zero.

enter image description here

enter image description here

What am I doing wrong here? I entered those expressions by double-clicking the gridcode fieldname in the calculator. (My goal is to use this within a stand-alone Python script but I'm debugging it here since the script isn't working either.) Both fields are definitely of type FLOAT.

4
  • Hi Stephen, try float(!GRIDCODE!)/100000.0 - I've had issues with the python calculator and integer division and found that floating the integer types help. Sep 8, 2014 at 4:55
  • Thanks, I just reached the same conclusion while debugging in Python. inValue / 1000000 => 0 while float(invalue) / 1000000 => 0.03563. Reckon this is a bug, or a known issue? Sep 8, 2014 at 4:58
  • 2
    In Python however, integer division truncated the result to an integer - I guess it's a feature of python ;) PS put this as an answer and I'll accept it Sep 8, 2014 at 5:00
  • stackoverflow.com/questions/183853/… has an answer that says: "It helps to clarify for the Python 2.x line, / is neither floor division nor true division ... / is floor division when both args are int, but is true division when either or both of the args are float." My understanding is that Python 3.x works the way you originally expected.
    – PolyGeo
    Sep 8, 2014 at 5:02

1 Answer 1

3

As you've discovered python truncates integers that are divided - this may include floating point fields with integers stored in them, sometimes 'duck typing' isn't your friend!

To force the issue and produce a floating point answer from a division float your number first:

float(!GRIDCODE!)/100000.0 

It is best to do this always when dividing otherwise if your enumerator (value in the field) is an integer you could get a truncated answer thanks to pythons' sometimes handy duck typing of variables.

For example if most of your values are like 999.999999 but one is actually 1000 and you divide by 10000 you will get around 0.99999999999 as the answer for most and 0 for the one that has an integer value...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.