5

I’m trying to update some GPS point coordinates in an attribute table using QGIS Field Calculator. However, regardless of whether I do it by creating a new field or updating an existing one, I get about 13 decimal places, the last 5 of which are “9” for every feature. (The original points only had 6 decimal places.) I doubt that this is really a coincidence! It happens even if I stipulate a lower Output field width and Precision. My expression is $y, for example, for latitude.

Any idea what’s going on here?

4
  • I assume you're seeing the data is stored in binary problem where not all values can be precisely stored. Some info here: stackoverflow.com/questions/618535/…
    – mkennedy
    Mar 13, 2013 at 16:09
  • Thanks, mkennedy, but I'm afraid that discussion is a bit beyond me. So do you have any idea what I can do to make my data look reasonable?
    – TVZ
    Mar 13, 2013 at 17:32
  • Here's a clue I just discovered. My original data was NAD 83, so I filled in 2 columns w/ projected data. That went ok. Then I saved the layer as WGS and ran the same expression to get lat/long data. That's when I got all the 9's.
    – TVZ
    Mar 13, 2013 at 17:47
  • I had a similar problem here: gis.stackexchange.com/questions/108336/… May 22, 2015 at 7:26

1 Answer 1

1

That definitely sounds like an it's an issue with floating point representation.

Try using round($y, 6) instead of just $y to round to six digits after the decimal place.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.