5

I am running simple field calculator expressions and experiencing some of the silliest results I have ever seen.

I am working on census data (living as a shape file) and trying to perform a simple [fieldName]/[fieldName] calculation on a newly created field.

ArcGIS happens to enjoy playing around with me... and returns values 0 and 1 in no particular order/reason that I can really justify.

Both input fields from the field calculator expression are of double data types. I have created a new field of double data type as well. However, the problem persists with other data types, including long/short integer.

Is this a bug? Has anyone else experienced this? Shortest route to create + update a new field other than moving between SPSS/Excel and ArcGIS all day long?

6

If one value or another can be rounded to an integer, the field calculator will format it as an integer and the operation (multiplication, division) will revert to integer arithmetic.

>>> 1/2
0

>>> 1/2.
0.5

>>> 1./2
0.5

>>> 1./2.
0.5

The trick here is to wrap them as floating points in the expression, do float([fieldName])/float([fieldName]).

  • They should be float types already since it was indicated that the fields were double in at least some cases. – user681 Feb 4 '12 at 11:59
  • 1
    But ensuring with an explicit float call should hopefully fix it, or at least act as a diagnostic to find the real underlying issue. – Jason Scheirer Feb 4 '12 at 21:19
  • This raises a geoprocessing error.. type mismatch 'float' – Michael Markieta Feb 5 '12 at 16:45
0

By not specifying the Precision and Scale of the new field, the calculation returned whole integers of 0 and 1. Creating a new field of double data type and Precision = 11 and Scale = 11 worked just fine.

However, if I were to create a copy of both [fieldName] in a new float data type field and perform the calculation, this solved the problem. Note that I didn't have to specify the precision or scale when doing so.

  • When creating a double field, you should specify a precision (aka field width) and scale (aka number of decimal places). I don't know what the defaults that arc uses but obviously they aren't sufficient. – user681 Feb 5 '12 at 19:35
  • 0 and 0 are the defaults, go figure! – Michael Markieta Feb 5 '12 at 20:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.