Skip to main content

Questions tagged [precision-scale]

Precision is the number of digits in a number. Scale is the number of digits to the right of the decimal point in a number. For example, the number 123.45 has a precision of 5 and a scale of 2.

Filter by
Sorted by
Tagged with
1 vote
0 answers
39 views

AGOL: Reduce precision in hosted feature layer view

I’d like to share an AGOL hosted feature layer view with a sister organization, but I want the locations of my records to be rounded to 3 decimal degrees accuracy in the view layer, so that exact ...
Nova's user avatar
  • 225
0 votes
0 answers
56 views

Is expressing XYTolerance in degrees an oxymoron?

Given: an ArcGIS featureclass whose spatialreference is a geographic coordinate system (GCS), and an XYTolerance expressed in decimal degrees. Find: a way to union (i.e. dissolve) adjacent polygons ...
Kirk Kuykendall's user avatar
1 vote
0 answers
258 views

How to preserve imported shapefile's original precision decimals in QGIS

How can I preserve an imported shapefile's original precision decimals in QGIS? Whenever I import an ESRI shapefile into QGIS, it automatically rounds off all fields containing decimals (data type ...
vlade113's user avatar
0 votes
1 answer
432 views

Export UTM coordinates with sub-meter precision from a shapefile in ArcGIS Pro?

I have a series of points in ArcGIS Pro that were collected with sub-meter precision. I need to export UTM coordinates for each point so that I can use them for some processing outside of Arc, and I ...
JuliaT's user avatar
  • 11
1 vote
1 answer
671 views

What is the practical limit for lat/long measurement accuracy?

There are several systems for finding lat/long coordinates. This includes GPS satellites, DoD TRANSIT, Russian GLONASS, Galileo, differential GPS, LORAN, land-based systems, and more. Maybe even using ...
William Entriken's user avatar
0 votes
0 answers
62 views

Shift in polygons due to precision loss

I am experiencing some shifting polygon that I obtain via pixel to lat lon conversion. I have a program to do image processing in opencv that uses pixel coordinates and once I had obtain my contours, ...
snookrun's user avatar
  • 243
1 vote
1 answer
266 views

Python toolbox field value precision - selection based on Double field

I have a python toolbox (.pyt) where the user selects a layer, a field from that layer and then values from that field. The tool uses this to make a selection based on the field values. It's working ...
jbosq's user avatar
  • 449
1 vote
0 answers
223 views

Reduce Z value precision in QGIS [duplicate]

The .asc LiDAR files provided by the environment agency have height values provided to a precision of the level used to measure atoms. How to quickly lose a few of the pointless digits on the end to ...
Sethinacan's user avatar
  • 1,724
2 votes
2 answers
203 views

Modifying vertices x,y values to remove all value from 3rd decimal in ArcMap?

I want to check whether there is a way to update the vertices x, y value and remove all values from the 3rd decimal. I only want 2 decimals (no matter the 3rd decimal is 0 or 9, just remove it). The ...
Maple Li's user avatar