0

Say that in an edit session I create a line of an exact length entered by keyboard - say 10'

If I later measure that line using the measure tool, snapping to each end of the line, it reads 9.99998'

Then when I check the Attribute Table immediately after completing the line, I find that the line length is 9.99998'.

This is an actual example I just tried, but it it seems to be consistent with all such measurements. They're all off - always less than what I entered as far as I can tell - by a small amount.

Now, I realize this is a very small difference, but shouldn't the editing tool build a line that is exactly 10' if that is the length I enter, and then shouldn't the measure tool "measure" the line to be 10' if that is the length I entered when creating the line?

What is going on here? And although for what I am likely to be doing it doesn't matter, is there a way to prevent this.

To clarify: I create the line in an ArcView edit session by clicking where I want the line to start, then right clicking and selecting "exact length" to type in the length of the line I want to create. I type in 10', and the next node is dropped, in theory at least, exactly 10' from the last point. Then I click F2 to complete the drawing. When I check the attribute table it shows that that the segment is 9.99998' rather than the 10' I specify. I first noticed when I was using the measuring tool - snapping at the beginning and the end of the line. I'm not exactly sure how I could clarify the more.

I am using ArcView 10.1 SP1 Data frame and data are both NAD_1927_StatePlane_New_York_Long_Island_FIPS_3104. Data are created and stored in a File Geodatabase. Not sure what the tolerances are at the moment.

  • 1
    Please edit the question to clarify how you are measuring length -- Are you doing heads-up digitizing or using the actual coordinates stored in the geometry? Also of critical importance: the storage format of the data and All the parameters of the coordinate reference (including the precision and tolerances). – Vince Apr 15 '16 at 20:26
  • @Vince - I create the line in an ArcView edit session by clicking where I want the line to start, then right clicking and selecting "exact length" to type in the length of the line I want to create. I type in 10', and the next node is dropped, in theory at least, exactly 10' from the last point. Then I click F2 to complete the drawing. When I check the attribute table it shows that that the segment is 9.99998' rather than the 10' I specify. I first noticed when I was using the measuring tool - snapping at the beginning and the end of the line. I'm not exactly sure how I could clarify the more. – That Idiot Apr 15 '16 at 23:21
  • Please edit the question in response to requests for clairifcation -- It's not fair to those who would have to answer to need to search comments for critical information. You haven't provided the geodata storage format, or the coordinate reference tolerance parameters, so there's still more you can do here. – Vince Apr 16 '16 at 0:14
  • -1 because GIS is not a tool to research things at molecular level. – FelixIP Apr 16 '16 at 9:51
  • 6.1 microns is short of molecular level, but it is smaller than a fog droplet – Vince Apr 16 '16 at 12:20
4

Lines and polygon dimensions are never exact and subject to the limitations of the feature class resolution and tolerance. Infinite precision is not possible and as a result these settings tell GIS to consider measures below those levels as "close enough" to exact measurements. There is no setting for absolutely no tolerance and infinite resolution for measurements, since the capacity of computers to store information is not infinite. Setting your tolerance and resolution to extremely small distances will reduce the potential limits of the area you can cover on the face of the globe compared to higher levels of tolerance and resolution, and may have consequences affecting performance, disk space use and snapping. This help topic provides more information about the effects of resolution and tolerance settings.

The computer has to fit the end points to a grid defined by the resolution and that takes precedence over the distance you entered. The help makes that clear. What you enter is less important than what a computer can actually efficiently store. The tool you are using to create your line will identify the grid cell that the end point of the line will fall in and then the line end point is snapped to the closest corner point of that cell. After it determines the end points it can actually store, then it recalculates the distance based on those points and reports it. The only time it might be exact is if your line was exactly aligned to the x or y plane, but probably not after you apply a projection that has to distort a 3D curved surface on to a 2D grid.

Another note is that in the background GIS is standardizing and storing all Projected Coordinate System linear positions and measurements relative to meters. All projected measurements based on English units will undergo unit conversions at some point during GIS processing. That is why when I set up my Linear Units as Foot_US in the projection I use it has to be expressed as 0.3048006096012192 in the XY Coordinate System (the equivalent of 1' expressed as a fractional portion of 1 meter). Add to this conversions between base 10 and base 2 for numbers to be manipulated and stored by the computer, and it should not be too surprising that 10' is not 10' after GIS gets through with it. The Projected Coordinate System itself probably has a larger impact on why there is distortion in the distance, but all of these complex processes and conversions can contribute some levels of variance between the user's initial input and the final output.

As a final note, a human hair is on average estimated to be about 0.1 mm thick in diameter, which is the equivalent of 0.000328 feet. So 0.00001 feet is approximately 33 times smaller than the width of an average human hair. Surveyors typically don't report distances less than 0.01 feet, which is approximately 30 times wider than an average human hair. So 0.00001 feet is a level of precision that is 1000 times finer than what is typically reported on most surveyed maps and documents. If that amount of measurement variance in the field is not negligible for your purposes, than I pity you for the amount you will have to pay for surveys.

  • But if I am creating a line segment by clicking a point, and then entering a distance to dictate how long the segment should be (in this case I typed 10' in the field for how long it should be), then why would the attribute table read something different? – That Idiot Apr 15 '16 at 23:15
  • 1
    The computer has to fit the end points to a grid defined by the tolerance and resolution and that takes precedence over the distance you entered. The help makes that clear. What you enter is less important that what a computer can actually efficiently store. After it determines the end points it can actually store then it recalculates the distance based on those points and reports it. The only time it might be exact is if your line was exactly aligned to the x or y plane, but probably not after you apply a projection that has to distort a 3D curved surface on to a 2D grid. – Richard Fairhurst Apr 16 '16 at 0:19
  • promote this to an answer and I'll accept it as the answer, although the measurements and readings were taken immediately after the creation of a horizontal line. – That Idiot Apr 16 '16 at 0:36
  • I don't think is possible to actually measure in the real world, to that kind of accuracy. Stating that something is more accurate than it is actually is, is wrong. You are introducing a false sense of accuracy. – nickves Apr 16 '16 at 23:35
  • I agree that reported measurements should be limited to distances of 0.01 feet for survey work, which is within the certifiable tolerances of current survey instrumentation and practices. The distance reported by GIS for the line in question would be 10.00 feet if that level of accuracy was observed. Exceeding that limit does not contribute any real value to the distance reported if the practices and instruments used to measure it cannot consistently achieve those levels of accuracy. That is why I added a description of how small the distance of 0.00001 feet is using a real world comparison. – Richard Fairhurst Apr 17 '16 at 1:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.