Using ArcGIS Pro, I classified a field of grain sorghum (supervises, pixel-based support vector machine) using 5 classes: Soil, Shadows, Leaves, Heads, and Ground targets (this was a UAS study). The spatial resolution of the input raster was about 2.1 cm/pixel, which is very high.
To test the accuracy of the image, you make a confusion matrix, which is a random assortment of test points in the pixel using ground-truthed data. The rule of thumb for this procedure is that you don’t want to use more than 10-times the number of test points than you have classes. Since I had 5 classes, each class of test points had 50 points. These points were randomly distributed throughout the image to avoid bias.
The problem that I’m having is that using those 50 points per class, I could not get a kappa value of below 1. Which indicates a very fine accurate classification, but since there is some error, I do want to detect it. To do this, I increased the number of points per class to 300; with this, I was able to compute a kappa value of 0.98, indicating a very high accuracy. These points were also randomly distributed. It should be noted that I placed them myself on the image.
My question is this: Is it ok to use more than the recommended “10x the number of classes” guideline to get test points for a confusion matrix?