Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am running simple field calculator expressions and experiencing some of the silliest results I have ever seen.

I am working on census data (living as a shape file) and trying to perform a simple [fieldName]/[fieldName] calculation on a newly created field.

ArcGIS happens to enjoy playing around with me... and returns values 0 and 1 in no particular order/reason that I can really justify.

Both input fields from the field calculator expression are of double data types. I have created a new field of double data type as well. However, the problem persists with other data types, including long/short integer.

Is this a bug? Has anyone else experienced this? Shortest route to create + update a new field other than moving between SPSS/Excel and ArcGIS all day long?

share|improve this question

If one value or another can be rounded to an integer, the field calculator will format it as an integer and the operation (multiplication, division) will revert to integer arithmetic.

>>> 1/2

>>> 1/2.

>>> 1./2

>>> 1./2.

The trick here is to wrap them as floating points in the expression, do float([fieldName])/float([fieldName]).

share|improve this answer
They should be float types already since it was indicated that the fields were double in at least some cases. – Dan Patterson Feb 4 '12 at 11:59
But ensuring with an explicit float call should hopefully fix it, or at least act as a diagnostic to find the real underlying issue. – Jason Scheirer Feb 4 '12 at 21:19
This raises a geoprocessing error.. type mismatch 'float' – Michael Markieta Feb 5 '12 at 16:45
up vote 0 down vote accepted

By not specifying the Precision and Scale of the new field, the calculation returned whole integers of 0 and 1. Creating a new field of double data type and Precision = 11 and Scale = 11 worked just fine.

However, if I were to create a copy of both [fieldName] in a new float data type field and perform the calculation, this solved the problem. Note that I didn't have to specify the precision or scale when doing so.

share|improve this answer
When creating a double field, you should specify a precision (aka field width) and scale (aka number of decimal places). I don't know what the defaults that arc uses but obviously they aren't sufficient. – Dan Patterson Feb 5 '12 at 19:35
0 and 0 are the defaults, go figure! – Michael Markieta Feb 5 '12 at 20:05

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.