0

I'm supposed to validate my supervised classification with a Confusion Matrix, but I'm having a hard time getting what I should use as a comparison.

At first, I used the same ROIs I created for classification to generate random samples throughout the image and then I reviewed it, making sure the pixels actually belonged to the different classes (that's what I called my "truth"). Thing is, my teacher said that wasn't ok and that there are known methods of obtaining your "truth". I tried to find it, but I just don't know the correct keywords to use on my searches. "Confusion Matrix" returns generic results explaining how to read it.

3

You validation data should be different from your training data. Otherwise you risk overfitting your classification model, and thus overestimating your accuracy.

In general terms, you would have your validation data from a higher resolution dataset, or even a field excursion, but it doesn't sound feasible in this situation. This is done in order to have better information than what is available in the classification itself and thus to have higher certainty in the validation.

Given that you most likely do not have such data available, you should randomly sample (using any of the many tools for creating random points) the entire area. Potentially, you could use your classification as input to the sampling tool, in order to get the same number of validation points for each class, in an effort to make the validation statistically significant.

All in all, the following list indicates the preferred order of approaches, with the best at top:

  1. High density systematic field survey.
  2. Lower density planned field survey (using the classification and satellite imagery for planning).
  3. Lower density random / unplanned field survey (high risk of a statistically invalid validation, due to low number of points in some of the classes).
  4. High density validation based on randomly sampled points in higher resolution satellite imagery.
  5. Lower density targeted validation (X number of points in each classification class) based on higher resolution satellite imagery.
  6. High density validation based on randomly sampled points in the same satellite imagery as the classification.
  7. Lower density targeted validation (X number of points in each classification class) based on the same satellite imagery as the classification.
  8. Using the training data as validation data (this should not be done)

As may be noted from the above list, the order of options is also inversely correlated with the effort related to implementing the options. As such, you need to weight your needs against the effort involved in doing the validation - while also explaining your reasoning behind the choices made.

  • I see. I appreciate your time for writing such a detailed, in-depth answer. Random sampling was my choice at a first moment (as explained in the question) but I used the training areas to provide input for the sampling tool instead of the classification, as you suggested. I'll try doing that as you described, but another method also crossed my mind. What do you think of drawing validation areas and applying region growing on them? – Eric Lino Nov 10 '16 at 1:08
  • @EricLino That doesn't really solve the problem with correlation between your training areas and your validation areas, which is the main concern. The two areas should be completely independent. – Mikkel Lydholm Rasmussen Nov 10 '16 at 9:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.