5

I am running simple field calculator expressions and experiencing some of the silliest results I have ever seen.

I am working on census data (living as a shape file) and trying to perform a simple [fieldName]/[fieldName] calculation on a newly created field.

ArcGIS happens to enjoy playing around with me... and returns values 0 and 1 in no particular order/reason that I can really justify.

Both input fields from the field calculator expression are of double data types. I have created a new field of double data type as well. However, the problem persists with other data types, including long/short integer.

Is this a bug? Has anyone else experienced this? Shortest route to create + update a new field other than moving between SPSS/Excel and ArcGIS all day long?

Improve this question

2 Answers 2

Reset to default
6

If one value or another can be rounded to an integer, the field calculator will format it as an integer and the operation (multiplication, division) will revert to integer arithmetic.

>>> 1/2
0

>>> 1/2.
0.5

>>> 1./2
0.5

>>> 1./2.
0.5

The trick here is to wrap them as floating points in the expression, do float([fieldName])/float([fieldName]).

Improve this answer
3
  • They should be float types already since it was indicated that the fields were double in at least some cases.
    – user681
    Commented Feb 4, 2012 at 11:59
  • 1
    But ensuring with an explicit float call should hopefully fix it, or at least act as a diagnostic to find the real underlying issue.
    – Jason Scheirer
    Commented Feb 4, 2012 at 21:19
  • This raises a geoprocessing error.. type mismatch 'float'
    – Michael Markieta
    Commented Feb 5, 2012 at 16:45
0

By not specifying the Precision and Scale of the new field, the calculation returned whole integers of 0 and 1. Creating a new field of double data type and Precision = 11 and Scale = 11 worked just fine.

However, if I were to create a copy of both [fieldName] in a new float data type field and perform the calculation, this solved the problem. Note that I didn't have to specify the precision or scale when doing so.

Improve this answer
2
  • When creating a double field, you should specify a precision (aka field width) and scale (aka number of decimal places). I don't know what the defaults that arc uses but obviously they aren't sufficient.
    – user681
    Commented Feb 5, 2012 at 19:35
  • 0 and 0 are the defaults, go figure!
    – Michael Markieta
    Commented Feb 5, 2012 at 20:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.