6

I recently calculated kappa (using confusionMatrix{caret} in R) between a thematic raster dataset at 1m spatial resolution and another at 30m spatial resolution. I was interested in quantifying the level of agreement between the two thematic raster datasets. Random points were taken at the same location for each thematic raster and tabulated. Thematic classes range from 1 - 6, which actually relates to the level of canopy cover (1 (low) - 6 (high)). The following is the resulting error matrix:

     30m
1m    1    2    3    4    5    6
1    9906  21   0    0    1    3
2     13   36   0    0    0    0
3     0    0    0    0    0    0
4     0    0    0    0    0    0 
5     0    0    0    0    0    0
6     0    0    0    0    0    0

Strangely, the overall accuracy and kappa seem to be at odds with each other:

Accuracy = 0.9962
Kappa    = 0.6649

What might be causing this discrepancy between Kappa and overall accuracy? My best guess is that the confusion matrix is stacked toward the lower canopy cover ranges (1 and 2) while there are a few far-off errors at the higher canopy cover classes (3 - 6).

  • 3
    (+1) This seems a misleading and inappropriate way to make the comparison. When the 30m cell has an intermediate (or even high) value, it could be comprised of 1m subcells exhibiting the full spectrum of values, whereas when the 30m cell has a low value, necessarily most of the subcells will have to have low values. Instead you need to compare the full distribution (of all 900 1m values) within randomly selected 30m cells. Even better, stratify the sampling to study the situation separately for each of the six levels of the 30m cells. – whuber Aug 12 '14 at 14:02
  • 1
    This is great @whuber, thanks. I agree that a stratified sampling design is much better. In fact, now that I've stratified the design, efficiency is significantly increased due to some tweaks to the script. – Aaron Aug 12 '14 at 15:53
7

kappa does not quantifies the level of agreement between two datasets. It represents the level of agreement of two dataset corrected by chance.

The reason why you have a large difference between kappa and overall accuracy is that one of the classes (class 1) accounts for the large majority of your map, and this class is well described. Overall accuracy is therefore an optimistic index of the classifier performance, even if it is the true "agreement" in your case. As a trivial example, if I give you a map that says "class 1" everywhere, it will be 99% correct. Similarly, if 99% of the pixels are randomly assigned to "class 1", the resulting map will still have a large agreement with your map. This is what kappa penalize with its "c" in the expression below (note that there are different kappa's, here is the most common).

kappa = (OA-c)/(1-c), where e is the overall probability of random agreement

On your confusion matrix, you can see that classes 5 and 6 are always wrong and class 2 is not very reliable. This will have a large impact on your kappa index and this explains the large difference. The classifier is not better than chance for these classes.

As a remark, standard OA and kappa DO NOT take the distance between classes into account, so the fact that classes 5 and 6 are far off does not affect your results for any of those indices. Therefore, I suggest that you take advantage of the fact that your classes refer to quantities. The correlation between the two map could therefore make a good indicator. A confusion between 1 and 6 would then have more importance than a confusion between 1 and 2. Another way is to look at each class individually (user and producer accuracies).

I do not agree on the fact that Kappa is largely considered to be more robust than OA. According to Pontius (2011), kappa has not provided the useful information that it is supposed to bring.

EDIT : More recently, Olofsson, Foody, Herold, Stehman, Woodcock and Wulder (2014, Remote sensing of Environment) also advocated against kappa. Considering the importance of those authors, I would follow their recommendations.

The problems associated with kappa include but are not limited to: 1) the correction for hypothetical chance agreement produces a measure that is not descriptive of the accuracy a user of the map would encounter (kappa would underestimate the probability that a randomly selected pixel is correctly classified); 2) the correction for chance agreement used in the common formulation of kappa is based on an assumption of random chance that is not reasonable because it uses the map marginal proportions of area in the definition of chance agreement and these proportions are clearly not simply random; and 3) kappa is highly correlatedwith overall accuracy so reporting kappa is redundant with overall accuracy.” (Foody, 1992; Liu et al., 2007; Pontius &Millones, 2011; Stehman, 1997). Consistentwith the recommendation in Strahler et al. (2006) the use of kappa is strongly discouraged as, despite its widespread use, it actually does not serve a useful role in accuracy assessment or area estimation.

  • This is very help, I appreciate the interpretation. – Aaron Aug 12 '14 at 13:45
  • 2
    Additionally, Pontius (2011) is a great read, however, the RS community has not fully embraced its message and still opts for kappa. Foody, G.M. (2009) is another monumental work pertaining to accuracy assessment: Classification accuracy comparison: Hypothesis tests and the use of confidence intervals in evaluations of difference, equivalence and non-inferiority. Remote Sensing of Environment, 113, 1658-1663. – Aaron Aug 12 '14 at 13:56
  • 2
    thanks for the extra link. It is true that there is no agreed upon alternative to Kappa, but I wanted to warn that kappa is not considered to be more robust than OA (I see that @Whiteboxdev's answer has been edited, so my answer could seem a bit extreme). Also, from what I know, G. Foody is not a defender of Kappa. See for instance Strahler et al (2006) landval.gsfc.nasa.gov/pdf/GlobalLandCoverValidation.pdf "(kappa) cannot be recommended as a general measure of map accuracy." – radouxju Aug 12 '14 at 19:29
3

The Kappa index of agreement (KIA) will tell you how much better, or worse, your classifier is than what would be expected by random chance. If you were to randomly assign cases to classes (i.e. a kind of terribly uninformed classifier), you'd get some correct simply by chance. Therefore, you will always find that the Kappa value is lower than the overall accuracy. The Kappa index is however considered to be a more conservative measure than the overall classification accuracy. Your KIA value is telling you essentially that your classifier is about 66% better than a random assignment of cases to the various classes. That's not bad!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.