# Is QGIS Raster calculator faulty?

I am trying to perform some rounding operations on rasters. I have used GDAL to convert the raster to an integer which is fine but now I want to /100 which should be simple in raster calculator but there seems to be some resampling or something going on, if I do 5431/100 I get 54.310001373291016 which doesn't make sense unless there is something I am missing. My Question is what could be causing this introduction of digits that I can't account for?

Qgis 2.14 OSX 10.11.1 KyngChaos

Sample file I am using

• The integer value gets obviously converted into 32-bit binary representation and 54.31 can't be expressed accurately with 32 bits. Try the calculator at babbage.cs.qc.cuny.edu/IEEE-754.old/Decimal.html Feed in 54.31, press the Rounded button and you'll get 54.310001 as 32-bit value. It is not right to say that numbers are random. Or do you get different result each time? Then I would call it random. Commented Apr 6, 2016 at 9:48
• As mentioned by @user30184, the values don't seem to be random. You can round the values if you would rather not have all the values after the decimal point by using Raster > Conversion > Translate as described in this post: How to round pixel values of a raster in QGIS? Commented Apr 6, 2016 at 9:51
• Yes I have managed to round the values as per that post, hence a value of 5431. But what if I want 1,2 or 3 decimals not 15 ? Commented Apr 6, 2016 at 10:02
• Apologies the numbers are not random but on the face of it they appear from no where Commented Apr 6, 2016 at 10:04
• User30184.00000000000154585 Would you like to put your comment as an answer as I think that is what I was missing. Commented Apr 6, 2016 at 10:10

## 1 Answer

The integer value gets obviously converted into 32-bit binary representation and 54.31 can't be expressed accurately with 32 bits. Try the calculator at babbage.cs.qc.cuny.edu/IEEE-754.old/Decimal.html Feed in 54.31, press the Rounded button and you'll get 54.310001 as 32-bit value.

Numbers do not change randomly when they are converted from decimals into 32 or 64 bit binary presentation and back. However, the conversion is not reversible for all decimal numbers. Closest 32 bit hex presentation of 54.31 is "42593D71". Conversion back to decimal yields 54.310001373291016. Next smaller hex value is "42593D70" and it yields 54.30999755859375.

Wikipedia article https://en.wikipedia.org/wiki/Single-precision_floating-point_format is quite comprehensive.