Does tf.keras.metrics.AUC work on multi-class problems?

Question

I have a multi-class classification problem and I want to measure AUC on training and test data.

tf.keras has implemented AUC metric (tf.keras.metrics.AUC), but I'm not be able to see whether this metric could safely be used in multi-class problems. Even, the example "Classification on imbalanced data" on the official Web page is dedicated to a binary classification problem.

I have implemented a CNN model that predicts six classes, having a softmax layer that gives the probabilities of all the classes. I used this metric as follows

self.model.compile(loss='categorical_crossentropy',
                       optimizer=Adam(hp.get("learning_rate")),
                       metrics=['accuracy', AUC()]),

and the code was executed without any problem. However, sometimes I see some results that are quite strange for me. For example, the model reported an accuracy of 0.78333336 and AUC equal to 0.97327775, Is this possible? Can a model have a low accuracy and an AUC so high?

I wonder that, although the code does not give any error, the AUC metric is computing wrong.

Somebody may confirm me whether or not this metrics support multi-class classification problems?

Answer 1

There is the argument multi_label which is a boolean inside your tf.keras.metrics.AUC call.

If True (not the default), multi-label data will be treated as such, and so AUC is computed separately for each label and then averaged across labels.

When False (the default), the data will be flattened into a single label before AUC computation. In the latter case, when multi-label data is passed to AUC, each label-prediction pair is treated as an individual data point.

The documentation recommends to set it to False for multi-class data.

e.g.: tf.keras.metrics.AUC(multi_label = True)

See the AUC Documentation for more details.

Answer 2

AUC can have a higher score than accuracy. Additionally, you can use AUC to decide the cutoff threshold for a binary classifier(this cutoff is by default 0.5). Though there are more technical ways to decide this cutoff, you could simply simply increase it from 0 to 1 to find the value which maximizes your accuracy(this is a naive solution and 1 recommend you to read this https://ncss-wpengine.netdna-ssl.com/wp-content/themes/ncss/pdf/Procedures/NCSS/One_ROC_Curve_and_Cutoff_Analysis.pdf for an in depth explanation on cutoff analysis )