From Wikipedia: In computing, floating point describes a method of representing an approximation to real numbers in a way that can support a wide range of values. The numbers are, in general, represented approximately to a fixed number of significant digits (the mantissa) and scaled using an ...

learn more… | top users | synonyms

1
vote
1answer
216 views

Floating point calculations in QGIS

How do I manage floating point calculations in QGIS? I have carried out 3 consecutive calculations on landsat bands. I end up with 32 bit floating point rasters on which I can't carry out a 4th ...
1
vote
0answers
77 views

Why does GeoServer's GetFeature round my numbers, but only for outputFormat=CSV?

When I make a GetFeature request with outputFormat=CSV to my GeoServer v2.4.5 I'm finding that the floating point numbers in my returned data have been rounded. When I make the same request with ...
0
votes
0answers
25 views

Simple query not returning values

I have a super simple select by attribute query "Coefficient" = #, the #'s are 0, .2, .3, etc... to 1. The query will select some of the #s but not others. Example) when I query 0,1,.5 the attributes ...