4

intermediate ArcMap user here. I'm trying to calculate the geometry for a shapefile such that I can add lat/long columns to my attribute table. While I have no trouble calculating the latitude, the longitude keeps coming out a decimal place off.

enter image description here

I know that the accurate longitude for this shapefile should be -74.067, -74.068, etc. I'm not sure if the negative sign is screwing up the significant digits? I looked around StackExchnage and no one seems to have this problem.

enter image description here

For reference, when calculating the Latitude, for "Type", I used "float" with precision = 6 and scale = 5. This worked fine for Latitude but still gives me the incorrect result for Longitude.

Also for reference: Projected Coordinate System: PCS_CarMAGBOG Projection: Transverse_Mercator False_Easting: 92334.87900000 False_Northing: 109320.96500000 Central_Meridian: -74.14659167 Scale_Factor: 1.00000000 Latitude_Of_Origin: 4.68048611 Linear Unit: Meter

Has anyone faced this problem before, and if so, how did you resolve it? Is the problem in calculate geometry or in the projection itself?

Improve this question
4
  • 1
    Have you verified that your calculation is working properly? I'm unsure of exactly what you mean by calculating the geometry, but my first thought is that the field values are exactly what is being calculated.
    – Evil Genius
    Commented Nov 21, 2013 at 15:44
  • @Evil Genius: Calculate Geometry is built in to ArcMap, right-click on a field name in the table view and you can fill that field with various geometric attributes of your features: length, area, etc.
    – Dan C
    Commented Nov 21, 2013 at 16:18
  • @DanC Thanks. I tried doing the same thing with some of my own data, which is much further north in USA LCC, but similar longitudes. Even specifying your same scale and precision, I get expected values back (ie -84.39175).
    – Evil Genius
    Commented Nov 21, 2013 at 16:29
  • Thank you so much for figuring this out! I was struggling with the same thing. Your suggestion above worked like a charm for me as well.
    – user53270
    Commented Jun 5, 2015 at 14:25

1 Answer 1

Reset to default
9

You should use a 'double' field type instead of 'float' as per Esri's recommendations. "When you create float and double fields and specify a precision and scale, if your precision is greater than 6, use a double" (http://resources.arcgis.com/en/help/main/10.1/index.html#//001700000047000000).

If you change your Precision and Scale to 8 and 5, respectively, then 'Calculate Geometry' in ArcMap will return the correct coordinates. Values of 7 and 5 will actually round the longitudes to integers. I believe the negative sign is treated counts towards the precision limit.

I downloaded your projection here (http://www.spatialreference.org/ref/sr-org/6873/) and actually tested this on a set of points I created with the same long/lat from your data. I could only get correct longitude calculations to work using the 8 precision and 5 scale.

Improve this answer
2
  • I've found that when using a double or float field, you do not need to specify scale & precision to calculate Lat/Long. In fact, this is only an option with shapefiles - "If the input table is a personal or file geodatabase the field precision/scale value will be ignored" (ESRI - same link as @Ryan S.posted)
    – JWallace
    Commented Jun 5, 2015 at 15:18
  • Double with Precision = 5 and Scale = 3 is the solution for similar inaccurate "Calculate Geometry" values for acres in ArcGIS 10.7.1
    – sirgeo
    Commented Dec 27, 2019 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.