Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [precision-scale]

Precision is the number of digits in a number. Scale is the number of digits to the right of the decimal point in a number. For example, the number 123.45 has a precision of 5 and a scale of 2.

Filter by
Sorted by
Tagged with
1
vote
1answer
172 views

What is the practical limit for lat/long measurement accuracy?

There are several systems for finding lat/long coordinates. This includes GPS satellites, DoD TRANSIT, Russian GLONASS, Galileo, differential GPS, LORAN, land-based systems, and more. Maybe even using ...
0
votes
0answers
30 views

Shift in polygons due to precision loss

I am experiencing some shifting polygon that I obtain via pixel to lat lon conversion. I have a program to do image processing in opencv that uses pixel coordinates and once I had obtain my contours, ...
1
vote
1answer
148 views

Python toolbox field value precision - selection based on Double field

I have a python toolbox (.pyt) where the user selects a layer, a field from that layer and then values from that field. The tool uses this to make a selection based on the field values. It's working ...
1
vote
0answers
173 views

Reduce Z value precision in QGIS [duplicate]

The .asc LiDAR files provided by the environment agency have height values provided to a precision of the level used to measure atoms. How to quickly lose a few of the pointless digits on the end to ...
2
votes
2answers
162 views

Modifying vertices x,y values to remove all value from 3rd decimal in ArcMap?

I want to check whether there is a way to update the vertices x, y value and remove all values from the 3rd decimal. I only want 2 decimals (no matter the 3rd decimal is 0 or 9, just remove it). The ...