Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

intermediate ArcMap user here. I'm trying to calculate the geometry for a shapefile such that I can add lat/long columns to my attribute table. While I have no trouble calculating the latitude, the longitude keeps coming out a decimal place off.

enter image description here

I know that the accurate longitude for this shapefile should be -74.067, -74.068, etc. I'm not sure if the negative sign is screwing up the significant digits? I looked around StackExchnage and no one seems to have this problem.

enter image description here

For reference, when calculating the Latitude, for "Type", I used "float" with precision = 6 and scale = 5. This worked fine for Latitude but still gives me the incorrect result for Longitude.

Also for reference: Projected Coordinate System: PCS_CarMAGBOG Projection: Transverse_Mercator False_Easting: 92334.87900000 False_Northing: 109320.96500000 Central_Meridian: -74.14659167 Scale_Factor: 1.00000000 Latitude_Of_Origin: 4.68048611 Linear Unit: Meter

Has anyone faced this problem before, and if so, how did you resolve it? Is the problem in calculate geometry or in the projection itself?

share|improve this question
1  
Have you verified that your calculation is working properly? I'm unsure of exactly what you mean by calculating the geometry, but my first thought is that the field values are exactly what is being calculated. –  Evil Genius Nov 21 '13 at 15:44
    
@Evil Genius: Calculate Geometry is built in to ArcMap, right-click on a field name in the table view and you can fill that field with various geometric attributes of your features: length, area, etc. –  Dan C Nov 21 '13 at 16:18
    
@DanC Thanks. I tried doing the same thing with some of my own data, which is much further north in USA LCC, but similar longitudes. Even specifying your same scale and precision, I get expected values back (ie -84.39175). –  Evil Genius Nov 21 '13 at 16:29

1 Answer 1

You should use a 'double' field type instead of 'float' as per Esri's recommendations. "When you create float and double fields and specify a precision and scale, if your precision is greater than 6, use a double" (http://resources.arcgis.com/en/help/main/10.1/index.html#//001700000047000000).

If you change your Precision and Scale to 8 and 5, respectively, then 'Calculate Geometry' in ArcMap will return the correct coordinates. Values of 7 and 5 will actually round the longitudes to integers. I believe the negative sign is treated counts towards the precision limit.

I downloaded your projection here (http://www.spatialreference.org/ref/sr-org/6873/) and actually tested this on a set of points I created with the same long/lat from your data. I could only get correct longitude calculations to work using the 8 precision and 5 scale.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.