1

I am following the example here to learn how to classify flood images. I am using a single band of the Sentinel-1 radar imagery. I am able to build and train my model without trouble.

The problem is when I test my model, I have a very low recall for the flood class.

Classification Results

Could someone explain why the recall is so low and how I could improve it?

4

Welcome to the trenches of classification, probability and statistics ;) .

Assuming you used sklearn, it has a detailed user guide on what metrics it provides to evaluate your classifications:

I'll quote the user guide on this:

Intuitively, precision is the ability of the classifier not to label as positive a sample that is negative, and recall is the ability of the classifier to find all the positive samples.

Let me elaborate on your example:
You have a fairly high precision for your flood class, so your model probably has a low risk of generating false positives. You have a low/average recall, so there's quite a bunch of samples that probably are flooded, but your model didn't classify them as flooded, false negatives. With a perfect model, both these values would be 1.0 . Sadly, the real world is horribly complicated. It's also why the real world is so amazing.

So, how does the classification know how bad or good your model is? If it's so smart and self-learning, why doesn't it just self-correct and present you results with a perfect score? Because the "perfect" results would then be mostly meaningless.

Recall is calculated as tp / (tp + fn) where tp is the number of true positives and fn the number of false negatives (Source). So it simply takes the values of the pixels you have carefully selected as ground truth, and compares how many of these pixels have also been classified by the model.

To improve your score, you have two options: Double-check your ground truth samples: If you have misclassified anything, that will throw off the score considerably. If you don't have actual ground truth information, and can only work off a satellite image, be very strict in your ground truth selection: Don't go near anything that seems less than "pure" flooded. If you are absolutely certain your ground truth is perfect, then you have to think about tweaking the classification method parameters, or choosing a different method entirely that is more appropriate for your data.

The second approach is to tweak and tune your model to classify more of the ground truth samples correctly. This increase in recall usually means a tradeoff on the precision score, because you will start falsely classifying values who you have sampled as belonging to other classes. This is where model-making starts to become an art form: Trial&error and lots of creativity are usually the key ingredients for great classification models.

In the end, think about what inputs make sense for your classification, don't just hunt score values. These score values do NOT express how good or bad your model is! Don't make the mistake of chasing perfect metrics! There are classifications that by design have a lot of overlap (e.g. flooded, partially flooded, previously flooded, ...). Probability metrics are simply a tool to help you understand your classification model and its results. It's a lot more important you understand why your classifications is producing false positives or not catching all positives, and that you then can explain the difficulties of your model to your audience.

That is also where trial & error (and logging the results of tweaked input or parameters) comes in handy: By generating a variety of results and reflecting on the impact each "knob" has, you will better understand your classification.

  • 1
    This is an excellent answer to a very broad question! – Kersten Apr 13 '17 at 7:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.