Why does field calculator expression sometimes return a binary value (0 or 1)?

I am running simple field calculator expressions and experiencing some of the silliest results I have ever seen.

I am working on census data (living as a shape file) and trying to perform a simple [fieldName]/[fieldName] calculation on a newly created field.

ArcGIS happens to enjoy playing around with me... and returns values 0 and 1 in no particular order/reason that I can really justify.

Both input fields from the field calculator expression are of double data types. I have created a new field of double data type as well. However, the problem persists with other data types, including long/short integer.

Is this a bug? Has anyone else experienced this? Shortest route to create + update a new field other than moving between SPSS/Excel and ArcGIS all day long?

-

By not specifying the Precision and Scale of the new field, the calculation returned whole integers of 0 and 1. Creating a new field of double data type and Precision = 11 and Scale = 11 worked just fine.

However, if I were to create a copy of both [fieldName] in a new float data type field and perform the calculation, this solved the problem. Note that I didn't have to specify the precision or scale when doing so.

-
When creating a double field, you should specify a precision (aka field width) and scale (aka number of decimal places). I don't know what the defaults that arc uses but obviously they aren't sufficient. – Dan Patterson Feb 5 '12 at 19:35
0 and 0 are the defaults, go figure! – Michael Markieta Feb 5 '12 at 20:05

If one value or another can be rounded to an integer, the field calculator will format it as an integer and the operation (multiplication, division) will revert to integer arithmetic.

``````>>> 1/2
0

>>> 1/2.
0.5

>>> 1./2
0.5

>>> 1./2.
0.5
``````

The trick here is to wrap them as floating points in the expression, do `float([fieldName])/float([fieldName])`.

-
They should be float types already since it was indicated that the fields were double in at least some cases. – Dan Patterson Feb 4 '12 at 11:59
But ensuring with an explicit `float` call should hopefully fix it, or at least act as a diagnostic to find the real underlying issue. – Jason Scheirer Feb 4 '12 at 21:19
This raises a geoprocessing error.. type mismatch 'float' – Michael Markieta Feb 5 '12 at 16:45