1

I used Random Forest for land cover classification on the Sentinel-2 images in a forest. I have 3 classes: forest (0), bareland (1), and cloud (2). My variables include 4 bands (B2, B3, B4, and B8), NDVI and 28 GLCM variables (using the library glcm).

When I trained the Random Forest model in R, I got the out of bag error is very low (0.01%), see the image attached. So I expected this model will provide good predictions. However, when I used this model for Sentinel-2 in the same area but captured at different times, the results had very low accuracy. I do not know why this happens. I guess maybe because of the overfitting of using texture variable GLCM?

Call:
 randomForest(formula = extract_samples$samples16Mar_22_v3_balanced ~      
., data = extract_samples, mtry = 5, ntree = 1000) 
 Type of random forest: classification
                 Number of trees: 1000
No. of variables tried at each split: 5

OOB estimate of  error rate: 0.01%
Confusion matrix:

0    1   2 class.error
0 2000    0   0      0.0000000
1    0 5188   0      0.00000000
2    0    0 150     0.006622517
0

1 Answer 1

1

The problem is in the GLCM features. You are adding some kind of spatial autocorrelation between the samples. In the glcm package GLCM features are calculated from a moving window. As the window size increase, more pixels will be in that neighborhood. Now, imagine an extreme case when the moving window is as big as the image. In this case GLCM features (ej. first order statistics like mean or variance) will be very similar in all pixels of the image and therefore the training samples and the OOB samples will be very similar. For this reason the OOB error decreases almost to zero as the size of the moving window increases.

I do not recommend to use the OOB error as an accuracy metric in remote sensing problems. The OOB error is useful to know the convergence of the model. That is, knowing the minimum number of trees that ensures convergence. However, to evaluate the performance of the model I recommend using independent samples that were not seen during training. You can use techniques like random cross validation, spatial cross validation, hold out methods, etc. The confusionMatrix function of the caret package can help you to obtain accuracy metrics such as: OA, precision, recall, accuracy by class, confidence intervals, Cohen's Kappa, etc.

2
  • 1
    The OOB is indicative of model fit so, can be informative in that context. When looking at accuracy/performance, you want to look at cross validation methods. Sep 7 at 7:57
  • I am not able to give a mathematical explanation for this. If you need it, maybe CrossValidated, Mathematics or Data Science community can help you.
    – sermomon
    Sep 13 at 8:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.