Questions tagged [precision-scale]

Precision is the number of digits in a number. Scale is the number of digits to the right of the decimal point in a number. For example, the number 123.45 has a precision of 5 and a scale of 2.

Filter by
Sorted by
Tagged with
0
votes
1answer
53 views

Export UTM coordinates with sub-meter precision from a shapefile in ArcGIS Pro?

I have a series of points in ArcGIS Pro that were collected with sub-meter precision. I need to export UTM coordinates for each point so that I can use them for some processing outside of Arc, and I ...
1
vote
1answer
311 views

What is the practical limit for lat/long measurement accuracy?

There are several systems for finding lat/long coordinates. This includes GPS satellites, DoD TRANSIT, Russian GLONASS, Galileo, differential GPS, LORAN, land-based systems, and more. Maybe even using ...
0
votes
0answers
38 views

Shift in polygons due to precision loss

I am experiencing some shifting polygon that I obtain via pixel to lat lon conversion. I have a program to do image processing in opencv that uses pixel coordinates and once I had obtain my contours, ...
1
vote
1answer
162 views

Python toolbox field value precision - selection based on Double field

I have a python toolbox (.pyt) where the user selects a layer, a field from that layer and then values from that field. The tool uses this to make a selection based on the field values. It's working ...
1
vote
0answers
199 views

Reduce Z value precision in QGIS [duplicate]

The .asc LiDAR files provided by the environment agency have height values provided to a precision of the level used to measure atoms. How to quickly lose a few of the pointless digits on the end to ...
2
votes
2answers
171 views

Modifying vertices x,y values to remove all value from 3rd decimal in ArcMap?

I want to check whether there is a way to update the vertices x, y value and remove all values from the 3rd decimal. I only want 2 decimals (no matter the 3rd decimal is 0 or 9, just remove it). The ...