I have created a VB.Net Add-In, and am using IRow to get the value of fields in tables in ArcMap to use in a query.

But I am encountering a problem when getting the values of the SHAPE.area and SHAPE.len fields. The values returned using IRow.Value seem to be rounded from the actual value of the field (the value that would be used to select using the table's Select by Attributes).

So, for example, I can select a polygon using Select by Attributes and the query:

SHAPE.area =  50.775141666668361

But the following code doesn't work to produce a value to use to successfully query:

Dim value As New Double

value = pRow.Value(pRow.Fields.FindField("SHAPE.area"))

And it doesn't work because value = 50.7751416666684

I tried it with value as a Decimal and value as a String. I also wrote the Row.Value directly to notepad without first storing it in a variable, and in all cases the number returned is rounded to two decimals fewer than the actual value.

Is there a way to get the actual, non-rounded, value?

I can use a workaround if I need to, but I'd rather be able to get the actual value of the area and length fields.

  • 1
    This isn't a GIS query so much as a generic "failure to bracket floating-point comparisons" computer science issue. If you want a specific row, you should use a rowid query (integer equivalence isn't as flighty as floating-point). – Vince Oct 13 '14 at 16:13
  • 1
    Just curious, are those units in meters? Even if they're kilometers or miles I can't imagine any GIS-related situation where you'd require that level of precision anyway. – pvdev Oct 13 '14 at 16:29
  • What confuses me is that two ways to find the value do not produce internally consistent results. The 15-digit precision number is there waiting to be used by ArcMap when I Select by Attributes and Get Unique values, but the Row.Value method doesn't give me that. – Dan Jurgella Oct 13 '14 at 16:39
  • That's because it's not possible to represent that value in an 8-byte IEEE floating point binary. – Vince Oct 13 '14 at 16:47
  • ICoursey, you are right in that I don't need that precision (units are in feet), except that when querying the table, if the user wants to use the values in the area or length field, then I need to get those values in order to query. But, yeah, I can just change the query to use a rounded value, which I guess I will have to do. – Dan Jurgella Oct 13 '14 at 16:47

The best way to deal with this seems to be by following the advice Microsoft gives about how to compare two doubles: https://msdn.microsoft.com/en-us/library/ya2zha7s(v=vs.110).aspx

Here's how I ended up implementing it in my code for ArcMap:

If I want to select a line based on its SHAPE.len value I would set the "where clause" of the query to be:

"ABS(SHAPE.len - value) <= ABS(SHAPE.len * .0000000000001)"

This would return all the records for which the first 13 decimal places are the same as the first 13 decimal places of value.

As stated in the question above, value is what is returned from pRow.Value(pRow.Fields.FindField("SHAPE.len"))

And this is an issue in ArcMap because that value is not the actual value that ArcMap will be querying against, which is why the double comparison method is necessary.

I found that for all the data I have encountered so far 13 decimal places works perfectly, but I'm not sure if this will always be the case. It might be necessary to reduce the precision, but that's easy to do by taking out some zeros in the query and testing.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.