I would like to know how precision is handled in ArcGIS software. Is it using a real Computer Algebra System engine, an arbitrary precision library or do others tricks to handle precision ?

  • 1
    A 'double' floating-point value has a 56 bits mantissa, which provides for ~15 digits. Rounding error in both geographic and projected coordinates is likely to be in nanometers, while coordinate accuracy is likely to be no better than centimeter (and often 2-3 orders of magnitude worse). What is driving your concern for precision? Are you designing a new generation of CPU dies in UTM or WGS84? – Vince Sep 29 '15 at 10:41
  • You have two questions here. GIS SE policy is to have "One question per Question". Please edit the question to focus on either the Projection Engine (which is in the title and first question), or implementing your own library. You should include the results of your empirical research on the actual loss you are experiencing, and if you focus on PE, compare it to the values obtained through the Esti library. – Vince Sep 29 '15 at 10:55
  • 1
    Just for reference, you should know that, while ArcGIS uses the Projection Engine, precision is implemented in the geometry libraries, one level higher in the stack. There is a white paper descibing how the PE and geometry libraries interface. – Vince Sep 29 '15 at 11:04
  • @Vince It was my misunderstanding that placed "Engine" next to "Projection" in the title, and so I have removed it - thanks! That White Paper looks like it could constitute an answer to the question. – PolyGeo Sep 30 '15 at 1:22

The forerunner of the current implementation of coordinate management in geodatabases was first integrated with the rest of the Esri product line (Arc/Info workstation, ArcView, and ArcIMS) when the "SDBE" product was purchased in the mid-1990s. At the time, the "SDE" training classes where half 'C' programming and half SQL, and multiple days were spent working on the transformation from double-precision floating-point values to integer array storage. By the time SDE reached version 3.0, the shape management functions were turned inside-out, and all the details of coordinate integerization were hidden inside the SgShape library as accessed by SE_SHAPE functions. At 9.0, coordinate references were updated to permit the use of 64-bit integers, and at 9.2 ArcGIS Desktop made HIGH precision coordinate references the default.

The particulars are covered in depth by the Esri Understanding Coordinate Management in the Geodatabase whitepaper, but the basic concept is to perform all coordinate comparisons in integer space. To make these integers, the double-precision coordinate values (in four dimensions, X, Y, Z , and M) have a false origin subtracted, and then the values are divided by a scaling factor, and rounded to the nearest integer. To return the encoded coordinate to double-precision, you just need to multiply by the scalefactor and add back in the origin.

For example, a point at the Equator along the Prime Meridian {0,0}, could have -180 subtracted from X and -90 subtracted from Y, with a X/Y scalefactor of 1000, to produce integer location {180000,90000}. Precision takes a part when determining the scalefactor, since the location {-0.0004,0.0003} would also map to {180000,90000}.

In reality, you should not ever "crowd" the lower-left corner of your data envelope with an origin at the same value (the conventional value for geographic coordinate systems is now {-400,-400}), since the origin represents the corner of the NE quadrant where coordninates can reside, and buffer operations which cross over that origin invalidate the geometry.

Similarly, you should never make a scalefactor so small that your polygon data collapses into a single point. But you also need to be careful of making the scalefactor too large as well. Back in the old days, BASIC precision shapes were limited by 31 bits (positive signed long), which only gave ~2.1 billion discrete values, so specifying a scalefactor of 200,000,000 meant you could only map a ~10 degree square at that precision. Modern HIGH precision shapes use 64-bit integer arrays, so they have a 56-bit precision ceiling (the 64-bit floating point mantissa size).

While HIGH precision is sufficient to map the globe with a 400M scalefactor, using such large scalefactors comes at a price: The integer array is stored in the database using a difference compression algorithm, so values that are 0.0001 degrees apart are evaluated as being 400,000 integers apart, which requires 32 bits, while a similar shape with a 1M scalefactor would only be 1000 apart, requiring only 16 bits. The BLOBs stored in the database using the very-large default scalefactors generated by ArcGIS Desktop can capture sub-millimeter precision, but can be significantly larger than their custom-generated brethren. Since databases usually perform linearly based on the volume of data they must move, doubling the storage can have a significant performance cost.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.