Accuracy assessment in R - Calculation of User Accuracy

Question

I want to assess the accuracy of the land cover classes classified by supervised classification of Landsat images. As reference data I use the aerial photography.

I sampled the classified Landsat data and at the same validation point, I identified the land cover class from aerial photography. I.e., each of my verification points have two attributes: one from Landsat (landsat), second from the aerial photography (reference).

I want to calculate an error matrix (contingency table), and accuracy assessment (overall accuracy, user's producer's accuracy) from the set of verification points.

I found a package greenbrown and asbio to assess the accuracy of classification. My final results of user's and producer's accuracy are howver switched between two packages.

Please, which package correctly calculate the values for user and producer accuracy?

Reproductible example

library(asbio)

# create dummy data
landsat <- c(1, 1, 1, 2, 2, 2, 3, 4, 5, 5, 5, 1, 1, 1, 2, 2, 2, 3, 4, 5, 5, 3, 3, 2, 2)
reference <- c(1, 2, 1, 2, 2, 2, 3, 4, 2, 2, 5, 1, 2, 2, 2, 1, 2, 3, 4, 5, 5, 3, 3, 2, 2)

# calculate Kappa statistics
asbio::Kappa(landsat,reference)  # Kappa(class1, reference)

# check out Kappa results
$ttl_agreement
[1] 76

$user_accuracy
1     2     3     4     5 
75.0  58.3 100.0 100.0 100.0 

$producer_accuracy
1     2     3     4     5 
50.0  87.5 100.0 100.0  60.0 

$khat
[1] 68.1

$table
       reference
class1  1 2 3 4 5
      1 3 3 0 0 0
      2 1 7 0 0 0
      3 0 0 4 0 0
      4 0 0 0 2 0
      5 0 2 0 0 3

# ----------------------------------------------------------------------   
# make the same calculation with the greenbrown package
# ----------------------------------------------------------------------

library(greenbrown)  
library(strucchange)
library(raster)
library(Kendall)
library(plyr)
library(bfast)
library(zoo)

# calculate the contingency table
tab <- table(landsat, reference)   data

# let's see the tab
tab

           reference
landsat   1 2 3 4 5
        1 3 3 0 0 0
        2 1 7 0 0 0
        3 0 0 4 0 0
        4 0 0 0 2 0
        5 0 2 0 0 3

# calculate the accuracy assessement
greenbrown::AccuracyAssessment(tab)

                    1        2   3   4   5 Sum UserAccuracy
1                 3  3.00000   0   0   0   6         50.0
2                 1  7.00000   0   0   0   8         87.5
3                 0  0.00000   4   0   0   4        100.0
4                 0  0.00000   0   2   0   2        100.0
5                 0  2.00000   0   0   3   5         60.0
Sum               4 12.00000   4   2   3  25           NA
ProducerAccuracy 75 58.33333 100 100 100  NA         76.0

The user and producer's accuracy are switched between two packages ! Please, which calculation and estimation of the user's and producer's accuracy is correct?

Answer 1

The User's Accuracy is the reliability of the classes in the classified image. It is calculated as the fraction of correctly classified pixels with respect to all pixels classified as this class in the image. For instance, based on your sample data, the User Accuracy for:

UA_Class1 is : 3 / 3+3 = 50
UA_Class2 is : 7 / 7 + 1 = 87.5

Therefore, to answer your question, it would seem that the User and Producer's accuracies from the asbio package are switched.

For a thorough explanation of the Error (Confusion) Matrix and related calculations, I refer you to this excellent technical note by D. Rossiter. I have included an excerpt that formally defines the UA.

It is expressed from the point of view of the mapper. Looking across the rows (classes as mapped), an error of commission is said to occur when the mapper incorrectly mapped this class at a reference (ground truth) site where it does not exist. That is, the mapper ‘committed’ the error of over-mapping a class. This leads to a lower user’s ‘accuracy’Ci.