Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

I realize I posed this question, and got answers, several days ago (see (Field calculator error)) , but none seem to really be working, and so am posing again.

Below are two screenshots that show the fields with field properties, and another showing the field calculator and the equation with the "answer" field populated by 0s. I apologize for them not being next to each other or in line with the appropriate text, but the web formatting keeps overlaying them.

Here is an overview of the problem: I am trying to do a in field calculator, where I'm trying to populate a new field with the values from a fairly simple equation of one field divided by another (Cnt1 by Pop1 in this case).

Both fields that I am dividing are double, and the value, while small is not infinitesimal, approx 0.01765.

I have tried creating various types of new fields: double, long, float, and played around with the scale/precision/properties, but everytime it returns "0" values. I know the values will be very small, i.e .04 but it doesn't seem to be a decimal display problem as I tried running it with Pop1/.4 and it correctly returned a decimal.

I've tested dividing the Cnt1 by .5 and that gives an appropriate decimal value, so it's not the new field format; the one thing that seems really strange is when I add (in field calculator python version): !Cnt1!+!Pop1! (which is 38+218699) I get 2.187, however, I suppose this could be related to the field being classified as float with precision=4, scale=3?

enter image description here

I've also tried the suggestions of @blord-castillo of doing:

1.0* (Cnt1/Pop1)

and also 1.0* (int(Cnt1/Pop1)) But neither works, I'm at my wits end on this one, and would greatly appreciate any help!

enter image description here

share|improve this question
At least for a double, leave it at 0,0 for the precision and scale. I would check the number format setting for the field. –  mkennedy Apr 7 '13 at 6:28
It's hard to follow what error from what input you are up to in your two questions. Are you able to re-phrase this one to something along the lines of "when I ran this Python code in the Python window, I received this error", please? –  PolyGeo Apr 7 '13 at 6:55
Can you add to your post the properties of each input and output field (type) ? –  A.R Apr 7 '13 at 12:32
Could you post a screenshot of your attribute table so we have an idea of what you're working with? –  Aaron Apr 7 '13 at 13:13
possible duplicate of Field calculator error –  Aaron Apr 7 '13 at 13:15

1 Answer 1

up vote 3 down vote accepted

I just ran a test at ArcGIS 10.0 SP5.

You can see my results below.

enter image description here

To achieve this in a file geodatabase polygon feature class, I had field cnt1 as Short Integer, pop1 as Long Integer, and Stnd3 as Double.

In the Field Calculator I used:


float(!cnt1!) / float(!pop1!)

I then used the context menu of the Stnd3 field to access the Numeric... button on its Properties to set the Number of decimal places to 9.

Is this the result you are after?

share|improve this answer
wow! This is perfect, it is exactly what I'm looking for. So the question for me to learn from this is why? Can you help me understand why the fields each need to be formatted differently in order to get the calculation to work? –  David Meek Apr 7 '13 at 22:55
It's probably confusion over different syntaxes needed for different parsers (VB,Python) and data type (File & Personal Geodatabases, shapefiles) combinations. I stick with Python and File Geodatabases wherever possible. The other thing is to just use the Numeric... button to format how many decimal places and allow the File Geodatabase to just store Float/Double (whichever you need). I prefer to use the Python float function to convert integers rather than just multiplying by 1.0 (but I think that should have worked too). –  PolyGeo Apr 7 '13 at 23:06
update is apparently this is a problem with my machine or SP or something as this still isn't working. Had someone try it on another machine and same code worked. Hmmm –  David Meek Apr 8 '13 at 1:05
question that year sago solved many problems with arcgis calculations: what is your windows format (region and Language in Control Panel) ?? did anyone by mistake changed the Numbers format (open additional settings and 2-check the Numbers formats). I hope it helps –  A.R Apr 9 '13 at 13:04

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.