I wonder if anyone else has run into this problem; and I'm baffled as to its reason and solution.

Whenever I do an export of features that have large-number attributes (over 14 digits) to a shapefile, they seem to lose precision. ArcGIS presents them in a default format of e-notation if they are large, but it seems to lose the last few digits of the number, for example:

728400000032249 becomes 728400000032000

Any ideas?

  • 1
    Most large-integer fields I have ever seen are really strings of decimal digits in disguise, such as US FIPS codes. (Not too many integral numeric quantities encountered in this world exceed 10^14! Disk storage, in bytes, and trillion dollar financial transctions, in cents, are the main exceptions.) If this is the case with you, would it be much effort just to store your attribute as a string? – whuber May 13 '11 at 22:31
  • You bring up... an excellent point! I will try that. – Nathanus May 13 '11 at 23:21

From the ESRI forum ArcGIS Desktop - Data Management (General) forum

Precision for shapefiles

A quote & credit to whuber (posted in 2007)

Double precision floats maintain 52 bits of precision, regardless of the sign of the number. That translates to 52 * log(2) = 15.65 decimal places. Some exceptional tiny values--less than about 10^-308 in size--have less precision; these are said to be "denormalized." Otherwise, the precision is the same for all doubles; there is no "typically" about it.

Your "bunch of extra zeros" is correct when it is understood in binary. In base 10, it is rare for a conversion from singles to doubles to result in additional zeros.

Shapefiles have to use better than single precision, which has only 23 bits of precision (about 6.92 significant decimal digits). Typical earth coordinate systems, like UTM, can have coordinates extending into the many millions of meters: that would limit shapefiles to about one meter precision when using such systems. One meter is not precise enough for large scale work. Thus, ESRI had no real choice in the matter when they designed shapefiles to use floating point coordinates rather than scaled integer coordinates.

source: http://forums.esri.com/Thread.asp?c=93&f=986&t=234634

Solution - avoid the shapefile format (stick to geodatabase) for better precision or accept the 1 meter precision.

  • 1
    On a hunch I repeated the process into a geodatabase, and it worked out just fine, as you suggested. Thanks for the concise explanation. – Nathanus May 13 '11 at 20:40
  • Although this brings up another confusion. That native shapefiles (the original data) don't have an issue with the large numbers. It is only after exporting that I run into trouble. Why might this be? – Nathanus May 13 '11 at 21:04
  • Mapperz/whuber's answer isn't entirely correct in this instance because presumably these large numbers aren't coordinates, right? What it looks like is the dbf format deals with numbers as an ASCII string, up to 32 characters in length. When this is loaded into arcgis, it presumably converts them to float64. Then (and this is the vague part), arcgis appears to cast them to float32 when you export the data. I don't know if there's a way to tell arcgis to do anything clever with these attributes when exporting to shp... – MerseyViking May 13 '11 at 22:19
  • 2
    @Mersey you are right. My 2007 response concerned shapefile vertex coordinates, not attributes, which are stored in dBase format. That format is effectively an ASCII decimal encoding: it cannot store floats or doubles directly. The format was originally limited to strings of 20 ASCII characters in [ .+0-9-]; going beyond that may be possible, but is asking for trouble. I believe you're correct in presuming ArcGIS is making an internal conversion to double during export (which is an outrageous programming error IMHO, but oh well...). – whuber May 13 '11 at 22:27
  • 2
    I'm glad you managed to find a solution. What appears to happen is not that dbfs can't store doubles, but that arcgis incorrectly converts the data when its writing the dbf part of a shapefile. en.wikipedia.org/wiki/IEEE_754-1985 tells you more than you'll need to know about floats and doubles, but in essence floats use 4 bytes to store decimal numbers, but doubles use 8 bytes - twice the storage but hundreds of times the range and precision. The shape exporter was probably written ages ago when there was less support for doubles. – MerseyViking May 14 '11 at 10:16

You are hitting the limit of precision of 32-bit floating point numbers by the looks of it. The only solutions are to a) use 64-bit floats (which I suspect aren't supported in Shapefiles) or b) Create an attribute that stores the least significant digits, and scale the original value down by a few orders of magnitude, trimming off that which is stored in your new attribute, when exporting. How the value gets remultiplied and added later on depends on which software you're loading it in to.

No matter how you cut it, float32s will only ever have a maximum of 6 significant decimal figures of precision.


I agree with MerseyViking's answer,
There are a few things you can do to circumvent the problem.
Another format you could possibly use..
sdf uses sql-lite which supports blob.
Or just sql-lite
Otherwise you would need to use probably a proprietary format like esri fgdb, or a full blown rdbms to hold the blobs (binary long objects)


The answer given to this question had alternative but not the solution.

The solution is as follows:

  1. In the Table of Contents, right-click on the layer with the Attribute Table of interest and select: Properties - Fields.

  2. Suppose you are converting STFID to a numerical field called STFID_NUM. Then in the “Layer Properties” window that opens, scroll down to the STFID_NUM row, and in the “Number Format” column click on the gray button with 3 dots.

  3. In the “Number Format” window that opens, select Category = “Custom”, and select String Format = “0”. Click OK, and in the Layer Properties window click Apply and then OK.

  4. If you check the Attribute Table again, then the full digit number for STFID_NUM should now be displayed.

  • There are three other answers to this question, which are you referencing? You can use the share button beneath it to copy a link to that answer into yours. – PolyGeo Oct 1 '15 at 0:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.