3

I'm having some problems to understand the result of my confusion matrix. Here is my case:

I've run a classification (random forest) on a satellite image. To do so, I created 50 random points for training and 50 random points for validation for each class. There are 6 classes in total. The code I used to create the points for each one is:

# create points
points<-randomPoints(myraster, 100) 

#add projection
pointsB<-SpatialPoints(pointsB, crs("+proj=utm +zone=33 +datum=WGS84 +units=m   
+no_defs +ellps=WGS84 +towgs84=0,0,0")) 

#create a df with ID
newpoints<-data.frame(ID=1:100) 

# merge each point with an ID
pBdf<-SpatialPointsDataFrame(pointsB, newpoints) 

# Split df for training and validation
trainingB<-pBdf[1:50,]
testB<-pBdf[51:100,]

Once the point dataset is created for each class, I merge them:

trainlist<-list(trainingB,trainingR,trainingS,trainingBu,trainingO, trainingW)
trainingpoints<-do.call("rbind", trainlist) 

testlist<-list(testB,testR,testS,testBu,testO, testW)
testpoints<-do.call("rbind", testlist) 

The output for testpoints is:

class       : SpatialPointsDataFrame 
features    : 300 
extent      : 379895, 390455, 6166685, 6173075  (xmin, xmax, ymin, ymax)
coord. ref. : +proj=utm +zone=33 +datum=WGS84 +units=m +no_defs +ellps=WGS84     
+towgs84=0,0,0 
variables   : 1
names       :  ID 
min values  :  51 
max values  : 100 

After the points are created and my classification is finished, here is how I created my confusion matrix

# Extract at test points the value of the classification
prediction<-extract(classification_raster, testpoints)
prediction<-unlist(prediction)
predictiontable<-as.data.frame(prediction)

# using the same test points extract pixel values from the reference data    
test<-extract(raster_Referecence, testpoints)
test<-unlist(test)
testtable<-as.data.frame(test)

confusionMatrix(data=predictiontable$prediction, reference=testtable$test) 

When checking the predictiontable and the testtable, there are 50 points per class, however the confusion matrix output is:

          Reference
Prediction  1  2  3  4  5  6
         1 43  4  0  1  0  9
         2  4 28  5  0 20  6
         3  0  2 44  0  0  0
         4  2  1  0 49  0  3
         5  0 14  1  0 31  0
         6  1  1  0  0  0 31

As you can see some classes have only 33 points and others have 57. Should not it be 50 in total per row?

Any idea?

  • I'm finding it difficult to figure out what you're doing when you name a bunch of variables that haven't been defined like pointsB and all the training and test data. Typically for training and testing data I use kfold and set.seed. What library are you using? Could you create a minimum reproducible example? – GISKid May 4 '17 at 14:31
  • 1
    Why are you unlisting your prediction and test extract object? Unless they are a polygon/line object the result should be a vector, or a data.frame if the raster object is a stack/brick. You could create the validation object by using: obs.pred<- data.frame(observed = extract(raster_Referecence, testpoints), prediction = extract(classification_raster, testpoints)) I would note that it is not a good idea to withhold data in a RF model and this does not indicate performance. You are getting an internal cross-validation indicating fit and can perform a Bootstrap to indicate performance. – Jeffrey Evans May 4 '17 at 18:34
  • @JeffreyEvans yeah I noticed that looking at his code as well. A bit wacky, but it looks like he's dealing with both test and reference in the same wacky way, and I think his data is staying lined up, so I don't think that's where his confusion is coming from. – SeldomSeenSlim May 4 '17 at 21:35
  • 1
    It's actually the columns, not the rows, that "need" to add up to 50. The row sums correspondo to how many of the "reference" points were assigned to each class. – lbusett May 6 '17 at 10:41
2

So for tl;dr answer to your question, No.

Long answer:

The 33..57; your rowsums, these are your models results. Notice that your colsums do add up to 50/class (except the last two, but I assume that you've made a transposition error some where. 49, 51 is close enough).

This implies that as you stated previously, you took a sample of 50 of each of your classes at points with known class identity. So you have 50 units of reference data for each class. You've compared this with your model prediction for the same units of data. If your model was a perfect model with 100% accuracy and precision, your row sums and colsums would all add up to be 50, and only your major diagonal would be populated with values. But this is the real world, so your model is confusing some results

Lets look at how well your model does at predicting class two. Your model predicted that 63/300 points were class two. So overall, it overestimated the amount of class two you should be finding. However, it only found 28 of the 50 points it would have found if it were a perfect model. This implies that not only is it overestimating class two, it also lacks precision with regard to finding class two.

In summary, your results look fine, you just need a bit of help with the interpretation, and you should probably figure out why you don't have the right number of reference points in classes 5 and 6. Other than that, this looks like exactly what you should expect from the kind of classification you are conducting.

| improve this answer | |
  • Thanks a lot for your answer and the comments of the rest. I was confused about how to read properly the information of the confusion matrix – GCGM May 7 '17 at 9:38
  • So I didn' want to include this in the original answer, since it was slightly off topic, but you should watch this video: youtube.com/watch?v=fAfXirQ5UsE regarding error assessment in remote sensing exercises. You should seriously consider the possibility that your current method for taking a random sample for generating reference data is biased, and that the results of your error assessment are really valid (even though oodles of people use the identical methods for conducting an error assessment). – SeldomSeenSlim May 9 '17 at 16:36
  • Edit: ... *aren't... – SeldomSeenSlim May 9 '17 at 17:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.