4

I’m trying to update some GPS point coordinates in an attribute table using QGIS Field Calculator. However, regardless of whether I do it by creating a new field or updating an existing one, I get about 13 decimal places, the last 5 of which are “9” for every feature. (The original points only had 6 decimal places.) I doubt that this is really a coincidence! It happens even if I stipulate a lower Output field width and Precision. My expression is $y, for example, for latitude.

Any idea what’s going on here?

  • I assume you're seeing the data is stored in binary problem where not all values can be precisely stored. Some info here: stackoverflow.com/questions/618535/… – mkennedy Mar 13 '13 at 16:09
  • Thanks, mkennedy, but I'm afraid that discussion is a bit beyond me. So do you have any idea what I can do to make my data look reasonable? – TVZ Mar 13 '13 at 17:32
  • Here's a clue I just discovered. My original data was NAD 83, so I filled in 2 columns w/ projected data. That went ok. Then I saved the layer as WGS and ran the same expression to get lat/long data. That's when I got all the 9's. – TVZ Mar 13 '13 at 17:47
  • I had a similar problem here: gis.stackexchange.com/questions/108336/… – Piotr Sadowski May 22 '15 at 7:26
1

That definitely sounds like an it's an issue with floating point representation.

Try using round($y, 6) instead of just $y to round to six digits after the decimal place.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.