0

I'm a bit confused, and maybe someone can solve my problem of understanding.

I've classified some imagery and performed my accuracy assessment. While following the usage guideline for ee.FeatureCollection.ErrorMatrix() the final array containing the Error Matrix values seems to be transposed.

As row totals I'm getting the reference/validation totals and not as wanted the classification totals. I'm using the following statement: basescene_error_matrix = validation.errorMatrix(actual='validation', predicted='classification')

I'm aware of the possibilty of just switching my inputs for validation.errorMatrix(), however, this would mean, that the function parameters are inversed.

Attached you see an example error matrix I created using the command. However, the row total values match my validation sample size. Yet, they should be within the column totals.

My Validation samples sizes are as follows:

Class Sample size
Informal 45
Formal 43
Industrial 35
Roads 42
Vacant Land 100
Vegetation 53
Waterbodies 32

enter image description here

5
  • Without having more data to see, seems to be transposed. Since you are using python, use sklearn.metrics.confusion_matrix. Also, did you split train/test samples? Why are you using unbalanced sample sizes for your analysis? That could mess your classification model
    – aldo_tapia
    Commented Apr 13, 2023 at 16:23
  • Thanks @aldo_tapia! I will have a look at sklearn. Yes, I've split my reference data into training and validation. However, my validation data was collected after the classification was done. This allows me to use a different approach. I'm following a combined approach of equal and proportional sample distribution for validation samples. I'm using a total of 350 validation samples for 7 classes of which 210 are assigned equally and 140 proportionally to the size. Therefore I'm using 30 + x % of 140 per class as validation sample size. Commented Apr 13, 2023 at 16:29
  • Thanks for the clarification. If you want to compute precision/recall, you must use equal sample sizes for validation. It's as useful as confusion matrix. Take a look, it's also in sklearn metrics module
    – aldo_tapia
    Commented Apr 13, 2023 at 16:35
  • Thanks for the hint, IMO the traditional error matrix, is sufficient enough for my analysis. Circling back to my question: Would you recognize this as a bug in GEE's ee.FeatureCollection.ErrorMatrix() module or am I doing something wrong? :D Commented Apr 13, 2023 at 16:40
  • It's probably a bug. You can report it in the Google group of GEE or in the github repo. I use GEE for getting data, but I train classification models in python/R so I haven't play with confusion matrix in GEE before
    – aldo_tapia
    Commented Apr 13, 2023 at 16:58

1 Answer 1

1

Without a reproducible code example it is difficult to evaluate your error matrix. Here's a quick test comparing a simple example from the sklearn.metrix.confusion_matrix and the output from ee.FeatureCollection.errorMatrix:

y_true = ee.List([2, 0, 2, 2, 0, 1])
y_pred = ee.List([0, 0, 2, 2, 0, 2])
seq = ee.List.sequence(0,5)
fc = ee.FeatureCollection(seq.map(
    lambda s: ee.Feature(None, {
    "actual":y_true.get(s),
    "predicted":y_pred.get(s)
    })))
print(fc.errorMatrix("actual","predicted").getInfo())

Here's the same code in the code editor: https://code.earthengine.google.com/a7662e9e83cca48249fde7e3249e247c

The output is:

[
  [2,0,0],
  [0,0,1],
  [1,0,2]
]

which is the same output as the sklearn example.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.