I would like to know how you determine the f12, f21 parameters for use in the McNemar`s test when for example I have two confusion matrices such as below:

(where f12 = number of cells misclassified by A but not by B And f21 = number of cells misclassified by B but not by A)

enter image description here

In the example with the two confusion matrices from above, I have the totals but am having difficulty in correctly determining the matrix elements f11, f12, f21, f22 below.

enter image description here

  • In the cross-classification matrix, your row and column totals should be 695 and 357 with a total of 2104. sum( rowSums(cm) + colSums(cm)). I modified my answer to include aggregating the confusion matrices into a cross-comparison matrix as well as a cumulative binomial matrix. – Jeffrey Evans Oct 10 '17 at 16:52

McNemar's test is a test for paired proportions, I do not see how it applies to a multi-class confusion matrix. Commonly, it is applied to validate logistic models. You cannot hope to aggregate the entire confusion matrix into an 2x2 contingency matrix and expect a valid hypothesis test.

I suppose that you could iterate through classes, deriving an appropriate 2x2 matrix for each class, but this would only give you a measure of per-class accuracy and not performance of the model which, would lead to a biased evaluation of performance. You could look at the Bowker test, which is an asymptotic extension of the McNemar's test that allows for multiple classes.

However, I do not believe the Bowker test is appropriate here because it tests if off-diagonal cells are equal to the cell symmetric to it. A test that compares the diagonal, such as the Kappa, would be much more informative for a multi-class model.

If you really want to do this you can calculate the percent correctly classified for each model and compare these proportions. However, this is telling you very little about model performance because the global model (pcc) could be highly biased towards a single very high performing class where the other classes are performing poorly. Whereas the other model may have poorer overall accuracy but a more even error distribution, making it more desirable.

Here is an example in R. First, create confusion matrix for each model.

classes = c("crop","water","pasture","fallow","forest","urban")

crop=c(34,4,5,3,1,1); water=c(23,29,5,6,0,4); pasture=c(7,2,22,9,0,0)
fallow=c(23,4,5,75,1,0); forest=c(1,0,1,2,77,0); urban=c(1,2,0,0,0,10)

( A <- matrix(c(crop, water, pasture,fallow,forest,urban), nrow = 6, 
               ncol = 6, byrow=TRUE, dimnames=list(classes,classes) ) )

crop2=c(73,0,0,4,3,2); water2=c(9,36,2,11,0,6); pasture2=c(2,2,30,19,0,0);
fallow2=c(5,2,5,59,1,0); forest2=c(0,0,1,2,74,0); urban2=c(0,1,0,0,1,7)
( B <- matrix(c(crop2, water2, pasture2,fallow2,forest2,urban2), nrow = 6, 
               ncol = 6, byrow=TRUE, dimnames=list(classes,classes) ) )

Calculate percent correctly classified for each model

( pcc1 <- sum(diag(A))/sum(A) )
( pcc2 <- sum(diag(B))/sum(B) )

Coerce into an appropriate 2x2 proportions matrix and run the McNemar test.

( m <- matrix( c(pcc1, (1-pcc1), pcc2, (1-pcc2)), 
                 nrow = 2, ncol = 2, byrow=TRUE) )           

McNemar's Chi-squared test with continuity correction: McNemar's chi-squared = 0.25451, df = 1, p-value = 0.6139

To specifically create the cross-comparison matrix detailed by the OP, you can use the matrix diagonal to construct the values.

# f11 - number of cases with correct classification in both A and B 
f11 <- sum(diag(A)) + sum(diag(B))

# f12 - number of cases wrongly classified by A but correctly classified by B 
f12 <- sum(diag(B)) - ( sum(A) - sum(diag(A)) ) 

# f21 - number of cases correctly classified by A but wrongly classified by B 
f21 <- sum(diag(A)) - ( sum(B) - sum(diag(B)) ) 

# f22 - number of cases wrongly classified in both A and B
f22 <- ( sum(B) - sum(diag(B)) ) + ( sum(A) - sum(diag(A)) )  

# Build matrix                         
( cm <- matrix( c(f11,f12,f21,f22), nrow=2, byrow=TRUE,
                dimnames=list( c("correct","incorrect"), 
                               c("correct","incorrect"))) )

Alternately, we calculate binomial (true/false) confusion matrices for each class, on the cumulative values of the two matrices. The result is a list containing a confusion matrix for each class with all other classes treated cumulatively.

( cm <- A + B )       

agg_mat <- list()   
  for (i in 1:nrow(cm) ) {
    agg_mat[[i]] <- t( matrix(c(cm[i,i], sum(cm[i,]) - cm[i,i], sum(cm[,i]) - cm[i,i], 
                           sum(cm) - sum(cm[i,]) - sum(cm[,i]) + cm[i,i]), ncol=2) )    
names(agg_mat) <- rownames(cm)

We can now calculate McNemar's chi-squared test for each class.

for(i in 1:length(agg_mat)) {
  cat(paste0("McNemar's chi-squared for ", rownames(cm)[i]), 
      round(mcnemar.test(agg_mat[[i]])$statistic,digits=3), "\n")

Here we can sum the class-level confusion matrices into a single matrix. You can average the summed matrices however, the accuracy results will not differ thus, making it an unnecessary step.

binomial_cm <- agg_mat[[1]]  
  for(i in 2:length(agg_mat)) binomial_cm <- binomial_cm + agg_mat[[i]]    
    rownames(binomial_cm) <- c("correct", "incorrect")
    colnames(binomial_cm) <- c("correct", "incorrect")

# average matrices
( binomial_cm.ave <- round(binomial_cm / nrow(cm), digits=0) ) 
  • Hi @Jeffrey Thanks for the quick reply! The 2 x 2 matrix dimension is not really a problem because larger confusion matrices (as in the example above) can be collapsed to 2 x 2 focusing on the distinction between correct and incorrect class allocations, as you correctly point out. This is where I am having the problem in identifying these specific values using the test equation below: χ^2=(f12- f21 )^2/(f12*+ *f21) (Post1of2) – Barry Apr 8 '16 at 9:38
  • (Post2of2) @Jeffrey Whereby the matrix elements are defined as: f11 - number of cases with correct classification in both A and B f12 - denotes number of cases that are wrongly classified by A but correctly classified by B f21 - number of cases that are correctly classified by A but wrongly classified by B and f22 - number of cases with wrong classification in both A and B – Barry Apr 8 '16 at 9:39
  • I have added the 2 x 2 contingency matrix for the example above to the original question which might better explain what I`m after. Thanks again. – Barry Apr 8 '16 at 9:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.