I would like to know how you determine the f12, f21 parameters for use in the McNemar`s test when for example I have two confusion matrices such as below:
(where f12 = number of cells misclassified by A but not by B And f21 = number of cells misclassified by B but not by A)
In the example with the two confusion matrices from above, I have the totals but am having difficulty in correctly determining the matrix elements f11, f12, f21, f22 below.
McNemar's test is a test for paired proportions, I do not see how it applies to a multi-class confusion matrix. Commonly, it is applied to validate logistic models. You cannot hope to aggregate the entire confusion matrix into an 2x2 contingency matrix and expect a valid hypothesis test.
I suppose that you could iterate through classes, deriving an appropriate 2x2 matrix for each class, but this would only give you a measure of per-class accuracy and not performance of the model which, would lead to a biased evaluation of performance. You could look at the Bowker test, which is an asymptotic extension of the McNemar's test that allows for multiple classes.
However, I do not believe the Bowker test is appropriate here because it tests if off-diagonal cells are equal to the cell symmetric to it. A test that compares the diagonal, such as the Kappa, would be much more informative for a multi-class model.
If you really want to do this you can calculate the percent correctly classified for each model and compare these proportions. However, this is telling you very little about model performance because the global model (pcc) could be highly biased towards a single very high performing class where the other classes are performing poorly. Whereas the other model may have poorer overall accuracy but a more even error distribution, making it more desirable.
Here is an example in R. First, create confusion matrix for each model.
classes = c("crop","water","pasture","fallow","forest","urban")
crop=c(34,4,5,3,1,1); water=c(23,29,5,6,0,4); pasture=c(7,2,22,9,0,0)
fallow=c(23,4,5,75,1,0); forest=c(1,0,1,2,77,0); urban=c(1,2,0,0,0,10)
( A <- matrix(c(crop, water, pasture,fallow,forest,urban), nrow = 6,
ncol = 6, byrow=TRUE, dimnames=list(classes,classes) ) )
crop2=c(73,0,0,4,3,2); water2=c(9,36,2,11,0,6); pasture2=c(2,2,30,19,0,0);
fallow2=c(5,2,5,59,1,0); forest2=c(0,0,1,2,74,0); urban2=c(0,1,0,0,1,7)
( B <- matrix(c(crop2, water2, pasture2,fallow2,forest2,urban2), nrow = 6,
ncol = 6, byrow=TRUE, dimnames=list(classes,classes) ) )
Calculate percent correctly classified for each model
( pcc1 <- sum(diag(A))/sum(A) )
( pcc2 <- sum(diag(B))/sum(B) )
Coerce into an appropriate 2x2 proportions matrix and run the McNemar test.
( m <- matrix( c(pcc1, (1-pcc1), pcc2, (1-pcc2)),
nrow = 2, ncol = 2, byrow=TRUE) )
mcnemar.test(m)
McNemar's Chi-squared test with continuity correction: McNemar's chi-squared = 0.25451, df = 1, p-value = 0.6139
To specifically create the cross-comparison matrix detailed by the OP, you can use the matrix diagonal to construct the values.
# f11 - number of cases with correct classification in both A and B
f11 <- sum(diag(A)) + sum(diag(B))
# f12 - number of cases wrongly classified by A but correctly classified by B
f12 <- sum(diag(B)) - ( sum(A) - sum(diag(A)) )
# f21 - number of cases correctly classified by A but wrongly classified by B
f21 <- sum(diag(A)) - ( sum(B) - sum(diag(B)) )
# f22 - number of cases wrongly classified in both A and B
f22 <- ( sum(B) - sum(diag(B)) ) + ( sum(A) - sum(diag(A)) )
# Build matrix
( cm <- matrix( c(f11,f12,f21,f22), nrow=2, byrow=TRUE,
dimnames=list( c("correct","incorrect"),
c("correct","incorrect"))) )
Alternately, we calculate binomial (true/false) confusion matrices for each class, on the cumulative values of the two matrices. The result is a list containing a confusion matrix for each class with all other classes treated cumulatively.
( cm <- A + B )
agg_mat <- list()
for (i in 1:nrow(cm) ) {
agg_mat[[i]] <- t( matrix(c(cm[i,i], sum(cm[i,]) - cm[i,i], sum(cm[,i]) - cm[i,i],
sum(cm) - sum(cm[i,]) - sum(cm[,i]) + cm[i,i]), ncol=2) )
}
names(agg_mat) <- rownames(cm)
agg_mat
We can now calculate McNemar's chi-squared test for each class.
for(i in 1:length(agg_mat)) {
cat(paste0("McNemar's chi-squared for ", rownames(cm)[i]),
round(mcnemar.test(agg_mat[[i]])$statistic,digits=3), "\n")
}
Here we can sum the class-level confusion matrices into a single matrix. You can average the summed matrices however, the accuracy results will not differ thus, making it an unnecessary step.
binomial_cm <- agg_mat[[1]]
for(i in 2:length(agg_mat)) binomial_cm <- binomial_cm + agg_mat[[i]]
rownames(binomial_cm) <- c("correct", "incorrect")
colnames(binomial_cm) <- c("correct", "incorrect")
binomial_cm
# average matrices
( binomial_cm.ave <- round(binomial_cm / nrow(cm), digits=0) )
