2

I am trying to perform some rounding operations on rasters. I have used GDAL to convert the raster to an integer which is fine but now I want to /100 which should be simple in raster calculator but there seems to be some resampling or something going on, if I do 5431/100 I get 54.310001373291016 which doesn't make sense unless there is something I am missing. My Question is what could be causing this introduction of digits that I can't account for?

Qgis 2.14 OSX 10.11.1 KyngChaos

Sample file I am using

5
  • 1
    The integer value gets obviously converted into 32-bit binary representation and 54.31 can't be expressed accurately with 32 bits. Try the calculator at babbage.cs.qc.cuny.edu/IEEE-754.old/Decimal.html Feed in 54.31, press the Rounded button and you'll get 54.310001 as 32-bit value. It is not right to say that numbers are random. Or do you get different result each time? Then I would call it random.
    – user30184
    Commented Apr 6, 2016 at 9:48
  • As mentioned by @user30184, the values don't seem to be random. You can round the values if you would rather not have all the values after the decimal point by using Raster > Conversion > Translate as described in this post: How to round pixel values of a raster in QGIS?
    – Joseph
    Commented Apr 6, 2016 at 9:51
  • Yes I have managed to round the values as per that post, hence a value of 5431. But what if I want 1,2 or 3 decimals not 15 ?
    – Sethinacan
    Commented Apr 6, 2016 at 10:02
  • Apologies the numbers are not random but on the face of it they appear from no where
    – Sethinacan
    Commented Apr 6, 2016 at 10:04
  • User30184.00000000000154585 Would you like to put your comment as an answer as I think that is what I was missing.
    – Sethinacan
    Commented Apr 6, 2016 at 10:10

1 Answer 1

2

The integer value gets obviously converted into 32-bit binary representation and 54.31 can't be expressed accurately with 32 bits. Try the calculator at babbage.cs.qc.cuny.edu/IEEE-754.old/Decimal.html Feed in 54.31, press the Rounded button and you'll get 54.310001 as 32-bit value.

Numbers do not change randomly when they are converted from decimals into 32 or 64 bit binary presentation and back. However, the conversion is not reversible for all decimal numbers. Closest 32 bit hex presentation of 54.31 is "42593D71". Conversion back to decimal yields 54.310001373291016. Next smaller hex value is "42593D70" and it yields 54.30999755859375.

Wikipedia article https://en.wikipedia.org/wiki/Single-precision_floating-point_format is quite comprehensive.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.