I would like to know the precision for an ArcGIS raster stored within a file geodatabase (FGDBR format); I can't seem to find this information anywhere in their documentation (apologies if I have missed the obvious...). By precision, I mean the number of decimal places that are used to store the value for each grid-cell (not the coordinate values for spatial referencing). I know this is 6 for the GRID format, but can't find the equivalent info for the FGDBR format.

The reason for asking is that I get different results for exactly the same processing when I use FGDBR vs. file stored GRID raster datasets. I am interested in the cumulative sum over the entire raster grid; there is a difference of approx. 0.5% between the two methods (i.e. a result of 5301.316 when the analysis is carried out in a GDB, compared with 5323.825 in a file storage system).

EDIT: In addition, students have reported getting vastly different answers when using 64 vs 32 bit machines; I don't have the values for these though. I will add when I can repeat the results myself.

closed as off-topic by PolyGeo Sep 28 '16 at 10:35

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This problem cannot or can no longer be reproduced. Changes to the system or to the asker's circumstances may have rendered the question obsolete, or the question does not include a procedure to enable potential answerers to reproduce the same symptoms. Such questions are off-topic as they are unlikely to help future readers, but editing them to include more details can lead to re-opening." – PolyGeo
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    Your definition of precision is still unclear. Are you concerned with the position of the upper-left cell center? Grids use single precision, which is more than six significant digits (but may be less than six to the right of the decimal). Given it's a raster, I'd hardly think less than half the cell size would matter. – Vince Apr 3 '14 at 10:27
  • Hi Vince - sorry for not being clear. I mean the precision at which the value for each grid cell is stored, not the spatial precision. I will edit my original post to clarify. – Kate Apr 3 '14 at 10:34
  • 1
    The precision for pixel values depends on the pixel depth -- 1-bit, 4-bit, 8-bit signed, 8-bit unsigned,... Floating-point values are IEEE standard 4-byte (Float) or 8-byte (Double). Then you get to the compression algorithm -- lossy or lossless. If you chose a lossy compression algorithm, there isn't much room to complain about precision. – Vince Apr 3 '14 at 11:40
  • Thanks again. I used lossless for both, so am just trying to understand the reasons for the differences. It's not a complaint - I just want to understand how the software is working so I can understand the outputs I get. – Kate Apr 3 '14 at 12:30
  • 1
    I've worked out where the discrepancy comes from - I had been using the statistics summary to get the overall sum of the dataset, rather than using the zonal statistics tool. If I use zonal statistics, I get the consistent answers I was expecting. Thanks for all your time - very much appreciated. – Kate Apr 3 '14 at 12:48