Take the 2-minute tour ×
Geographic Information Systems Stack Exchange is a question and answer site for cartographers, geographers and GIS professionals. It's 100% free, no registration required.

Can anyone show how to calculate the RMSE (root mean square error) between following two rasters step-by-step and discuss on the obtained results' min and max values, and how to interpret it. Thanks a lot in advance!

 First raster (original, 2 by 2):
 1 2
 3 4

 Second raster (obtained, 2 by 2):
 2 2
 4 1
share|improve this question

1 Answer 1

up vote 9 down vote accepted

Calculation

  1. Subtract one raster from the other. (The direction of subtraction does not matter.)

    -1 0
    -1 3

  2. Square the result.

    1 0
    1 9

  3. Average the values.

    (1 + 0 + 1 + 9)/(1 + 1 + 1 + 1) = 11/4.

    (I wrote this in a suggestive way to show how missing-data cells can be handled if your GIS does not have this capability: Create an indicator grid with 1's where you have data and 0's elsewhere. Divide the sum of your grid by the sum of the indicator grid. In Spatial Analyst you can get the sums as focal sums.)

  4. Take the square root.

    Sqrt(11/4) = 1.66

Interpretation

This number is a measure of the typical cell-by-cell difference between the two grids. When the grids have hundreds of values or more (as most do), they do not exhibit huge extremes or outlying values, and the average difference is zero, then the standard rule of thumb for interpreting the rmse is:

  • About 2/3 of all the cells will differ by less than the rmse.

  • About 95% of all the cells will differ by less than twice the rmse.

  • It will be unusual to see differences more than three times the rmse.

In a grid of any size (e.g., a million cells), "unusual" still translates to several thousand cells: around a fraction of one percent of all of them.

In the example--which is trivially small--knowing there are 4 cells and the rmse is 1.66, we would think "about 2/3 -- say 2 or 3--of the cells agree to within 1.66. Probably all of them agree to within 2*1.66 = 3.32." The actual state of affairs, as we can see from the result of step (1), is that 3/4 of the cells agree to within 1.66 and all of them indeed agree to within 3.

When the grids vary wildly and exhibit huge ranges of values, you might mistrust the rules of thumb. From Chebyshev's inequality you still know that

  • No more than 1/4 of the cells differ by more than twice the rmse.

  • No more than 1/9 of the cells differ by more than three times the rmse.

  • In general, pick any number k equal to 2 or greater. No more than 1/k^2 of the cells differ by more than k times the rmse.

This is a universal rule, valid for any pair of grids, whereas the previous rule of thumb assumes the distribution of cell differences is roughly "bell shaped" without many extreme outliers.

Edit

The preceding interpretations assume you are comparing two grids intended to represent the same thing, up to measurement error, so that their average difference is zero (or near enough to it). When the average difference is appreciable (compared to the rmse), these interpretations are incorrect--but then it also rarely makes sense to use the rmse. Instead, one would (a) report the average difference and (b) subtract its square after step (3). This gives the mean square residual rather than the mean square difference. Its square root is the typical size of variations between the two grids relative to their average difference. With this caveat, the interpretation can use the same rules of thumb as before.

share|improve this answer
    
@whuber: thanks a lot!!! is there any book that describes this process as you do in detail? or paper? as a reference. once more again thanks a lot!!! i really do appreciate it. –  opl Jan 4 '11 at 13:41
    
@whuber: how would the calculation change if we have the following rasters, raster1 (3 by 3)={{1,2,-9999},{2,3,-9999},{4,5,-9999}}, and raster2 (3 by 3)={{2,2,-9999},{-9999,3,4},{-9999,-9999,-9999}}. where no data value is -9999. thanks a lot!!! –  opl Jan 4 '11 at 14:23
    
@opl The difference of the two equals {{-1,0,Null},{Null,0,Null},{Null,Null,Null}}. Therefore the squared difference equals {{1,0,Null},{Null,0,Null},{Null,Null,Null}} and the indicator equals {{1,1,Null},{Null,1,Null},{Null,Null,Null}}. Their sums are 1 and 3, respectively, giving a mean square of 1/3 and there an rms of Sqrt(1/3). –  whuber Jan 4 '11 at 15:59
    
@whuber: is it possible to say that resultant 'rmse' value will be in some specific range, such as let's say min 0 and max 10...? so that i can say that, if it is close to minimum value then it is better, and when it is closer to maximum value then it is deviating significantly, or the other way around. thanks in advance! –  opl Jan 4 '11 at 20:20
    
@opl Usually you cannot predict the rmse in advance unless you know something about how the grids are likely to vary. For instance, the metadata for each grid might supply a quantitative indication of their likely deviation from the truth. Say, each is a DEM and one has a stated vertical error of +-15 m and the other has a stated vertical error of +-20 m. Taking these as rough indications of standard deviation, we can combine them to estimate a relative error of Sqrt(15^2 + 20^2) = 25 m. In this case I would expect the rmse between the two to be around 25 m. –  whuber Jan 4 '11 at 20:28

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.