Calculate kappa score for multi label image classifcation

Question

I want to calculate kappa score for a multi label image classification problem.

I don't think sklearn supports this inherently because when i try this

import sklearn
sklearn.metrics.cohen_kappa_score(y_test, predictions) 

i get

ValueError: multilabel-indicator is not supported

anyone has suggestions on how to do this? My prediction matrix is of shape(845,8) with 0 and 1 in them. My ground truth matrix is of shape (845,8) with 0 and 1 in them

My labels look like this

[0,0,1,0,1,0,1,0]

thanks in advance.

#######EDIT

please provide relevant code.Alteast a small snippet.

Answer 1

Cohen's kappa by definition does not support multiple labels

This is because Cohen's kappa is given by K = (po - pe)/(1 - pe)

where po = (TP + FN)/(TP + TN + FP + FN)

and pe = (TN + FP)/(TP + TN + FP + FN)

As you can see it is calculated using Confusion matrix.You can construct a confusion matrix only for mutually exclusive labels(ex: [0,0,1,0,0,0,0,0]) and not for non exclusive labels(ex: [0,0,1,0,1,0,1,0])

If using Cohen's kappa is a priority i would recommend calculating it separately for each and every label and then averaging them.

Or else the best course of action here would be to use Krippendorff's Alpha

Krippendorff’s alpha (also called Krippendorff’s Coefficient) is an alternative to Cohen’s Kappa for determining inter-rater reliability.

reference links: wikipedia , statisticshowto

Although Krippendorff's Alpha isn't available in sklearn library you can use it using this package krippendorff

import krippendorff
reliability_data = [[prediction matrix],[ground truth]]
print(krippendorff.alpha(reliability_data=reliability_data))

print([prediction matrix].shape,[ground truth].shape)

(845,8) , (845,8)

hope this helps

Answer 2

Krippendorff's Alpha

You can NOT use Cohen's Kappa for a multilabel agreement. The solution is to use Krippendorff's Alpha.

However, Krippendorff's Alpha can be used with many different data types and you need to remember that you have nominal data (i.e. classes).

Code

You can use the krippendorff package found here: https://github.com/pln-fing-udelar/fast-krippendorff

Notice that the alpha() function has a parameter level_of_measurement, which by default is interval, so you need to set it to nominal!

import krippendorff

reliability_data = [[predictions], [y_test]]

print(krippendorff.alpha(reliability_data=reliability_data, 
                         level_of_measurement='nominal'))