7

We are doing a check of a length attribute against the Shape Length in all our line feature classes (maybe 40 feature classes and 200,000+ features).

In the length attribute we round the Shape Length to one decimal place, so a shape length of 1.234567 has a Length attribute value of 1.2

When doing our checks, however, we are having some flagged as incorrect even though when you look at them they look valid.

In digging deeper, I've found we're getting different values (in python vs other methods) from what should be the same geometry for the same feature.

In the attribute table value = 351.448527:

enter image description here

In SQL value = 351.448527155212:

SELECT
    [OBJECTID],
    [Length],
    SHAPE.STLength() AS ShapeLength,
    ROUND(SHAPE.STLength(), 1) AS ShapeLengthRounded
FROM
    [WWMain_evw]
WHERE OBJECTID = 14228

enter image description here

But in the python window (which is how I was doing the comparison) the value returned is different - 351.519772688:

with arcpy.da.SearchCursor("WWMain", ['OID@', 'SHAPE@', 'Length'], "OBJECTID = 14228") as cursor:
    for row in cursor:
        print "ObjectID = {}, Length = {}, Shape Length = {}, Rounded Shape Length = {}".format(row[0], row[2], row[1].length, round(row[1].length, 1))

Which returns

ObjectID = 14228, Length = 351.4, Shape Length = 351.519772688, Rounded Shape Length = 351.5

I was expecting the Rounded Shape Length here to be 351.4, and as this value matches the Length attribute it should pass, but due to the different shape length it's flagged as a fail.

What could be causing this difference in reported Shape Length, and how do I force my python window to return the real value?

Values returned:

  • Attribute table: 351.448527
  • SQL query: 351.448527155212
  • Python query: 351.519772688

The attribute table and SQL are close enough to call them the same number, but the value the python query returns is different.

All my feature classes are New Zealand Transverse Mercator. I would expect that STLength is same system as Shape@.


In response to comments:

It doesn't happen to all features, just a handful of the ones I've looked at. One feature class (the WWMain one I'm looking at for this question) had less than 10, out of a total of 15000 features.

If I use SHAPE@LENGTH in the cursor instead of SHAPE@, the result is the same.

Enterprise geodatabase is on SQL Server 2012 SP3 running on Windows Server 2012 R2.

The following are enabled on my feature classes:

  • Versioning
  • Geometric network
  • 3d datasets (although no z data recorded)
  • Replication
  • Geometry configuration keyword

enter image description here

Screenshot of my Domain/Resolution/Tolerance tab. Resolution set to 0.0001 so differences of 70mm+ (0.07m) are way too big (in my opinion). I believe these are the default values (i.e. nothing was changed here on feature class creation).

enter image description here

From the comments I am starting to feel this is a bug or data issue rather than an expected or observed result. I'll log with Esri.

As requested - WKT (via cursor and SHAPE@WKT) of the same line I'm testing with above:

MULTILINESTRING Z ((1877081.4539999999 5818516.2470000004 0, 1877083.8540000003 5818515.9460000005 0, 1877083.665 5818514.3059999999 0, 1877089.5860000001 5818513.5350000001 0, 1877091.28018717 5818513.2565705059 0, 1877092.9907315627 5818513.10881723 0, 1877094.7075689049 5818513.0926095014 0, 1877096.4205978976 5818513.2080426821 0, 1877098.1197396496 5818513.4544376023 0, 1877099.7949969766 5818513.8303445559 0, 1877101.4365132232 5818514.3335518315 0, 1877103.0346302546 5818514.961098725 0, 1877104.5799452828 5818515.7092929604 0, 1877106.0633661889 5818516.5737324115 0, 1877107.4761650192 5818517.5493310047 0, 1877108.8100293358 5818518.6303486433 0, 1877110.0571111257 5818519.8104249788 0, 1877111.2100729754 5818521.0826168368 0, 1877112.2621312412 5818522.4394390639 0, 1877113.2070959632 5818523.8729085699 0, 1877114.0394072835 5818525.3745912975 0, 1877114.7541681603 5818526.9356518472 0, 1877115.3471731783 5818528.546905458 0, 1877115.8149332937 5818530.1988720512 0, 1877116.1546963621 5818531.8818320045 0, 1877116.3644633305 5818533.5858833445 0, 1877116.443 5818535.301 0, 1877117.716 5818551.3609999996 0, 1877136.7759999996 5818550.8600000003 0, 1877136.9484456284 5818552.2929528747 0, 1877137.2501382942 5818553.7043610262 0, 1877137.6785914272 5818555.0825915392 0, 1877138.2302736894 5818556.4162849514 0, 1877138.9006380807 5818557.6944488753 0, 1877139.6841594153 5818558.9065486044 0, 1877140.5743798607 5818560.0425939355 0, 1877141.5639621632 5818561.0932215098 0, 1877142.6447501229 5818562.0497719906 0, 1877143.8078358171 5818562.904361425 0, 1877145.0436330198 5818563.6499462361 0, 1877146.3419562115 5818564.2803812632 0, 1877147.6921045308 5818564.7904704232 0, 1877149.082949969 5818565.1760095283 0, 1877150.5030290901 5818565.4338209406 0, 1877151.9406375121 5818565.5617797626 0, 1877153.3839263751 5818565.5588313481 0, 1877154.8210000005 5818565.4249999998 0, 1877342.6270000003 5818550.4879999999 0, 1877346.6900000004 5818567.7199999997 0, 1877375.7400000002 5818586.7199999997 0))

One thing I did note when pulling out the WKT is that the line has arcs - I will need to test the other problem features to see if those also have arcs.


Additional discovery

If I specify the actual field SHAPE.STLength() I get the result I see elsewhere - 351.448527155

with arcpy.da.SearchCursor("WWMain", ['SHAPE.STLength()']) as cursor:
    for row in cursor:
        print row[0]

Result:

351.448527155
  • I ran a small experiment, exporting line vertices to points and recreating polyline by truncating coordinates after 2nd decimal. Gives about the same mismatch. Are there any settings in SQL thing playing this game? – FelixIP Oct 4 '16 at 3:01
  • @FelixIP I'm unsure whether it's something that SQL is doing, or even where to look – Midavalo Oct 4 '16 at 4:36
  • What coordinate system? Is Shape.STLength using the same coordinate system as Shape@? – Fezter Oct 4 '16 at 4:47
  • I'm not too sure, but I know there are differences between geography calculations and geometry calculations. – Fezter Oct 4 '16 at 4:54
  • As @Fezter pointed float type might not be able to handle 7 numbers before decimal point plus enough numbers after to return accurate length for nzrm2000. – FelixIP Oct 4 '16 at 5:26
1

I suspect this has to do floating point numbers (doubles) and how they are stored in the database and then represented using the "various methods". I am not sure if ESRI uses the IEEE 754 standard itself for storing storing floating-point numbers but it is widely used. If not IEEE 754, ESRI does do some approximation however.

Quoting ArcGIS Support:

"Floating point numbers are approximations that cannot precisely mimic real numbers, and floating-point operations cannot exactly mimic true arithmetic operations."

"If several arithmetic operations are performed on floating point numbers to obtain a given result, these rounding errors may be cumulative."

source: http://support.esri.com/technical-article/000010968

  • Thanks Jakub - that is what I think the cause must be, it just seems odd that I get different lengths from different query methods performed within ArcMap. – Midavalo Oct 5 '16 at 20:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.