(An update to this question has been added.)
I am a graduate student at the university of Ghent, Belgium; my research is about emotion recognition with deep convolutional neural networks. I'm using the Caffe framework to implement the CNNs.
Recently I've run into a problem concerning class imbalance. I'm using 9216 training samples, approx. 5% are labeled positively (1), the remaining samples are labeled negatively (0).
I'm using the SigmoidCrossEntropyLoss layer to calculate the loss. When training, the loss decreases and the accuracy is extremely high after even a few epochs. This is due to the imbalance: the network simply always predicts negative (0). (Precision and recall are both zero, backing this claim)
To solve this problem, I would like to scale the contribution to the loss depending on the prediction-truth combination (punish false negatives severely). My mentor/coach has also advised me to use a scale factor when backpropagating through stochastic gradient descent (sgd): the factor would be correlated to the imbalance in the batch. A batch containing only negative samples would not update the weights at all.
I have only added one custom-made layer to Caffe: to report other metrics such as precision and recall. My experience with Caffe code is limited but I have a lot of expertise writing C++ code.
Could anyone help me or point me in the right direction on how to adjust the SigmoidCrossEntropyLoss and Sigmoid layers to accomodate the following changes:
Thanks in advance!
I have incorporated the InfogainLossLayer as suggested by Shai. I've also added another custom layer that builds the infogain matrix H
based on the imbalance in the current batch.
Currently, the matrix is configured as follows:
H(i, j) = 0 if i != j H(i, j) = 1 - f(i) if i == j (with f(i) = the frequency of class i in the batch)
I'm planning on experimenting with different configurations for the matrix in the future.
I have tested this on a 10:1 imbalance. The results have shown that the network is learning useful things now: (results after 30 epochs)
These numbers were reached at around 20 epochs and didn't change significantly after that.
!! The results stated above are merely a proof of concept, they were obtained by training a simple network on a 10:1 imbalanced dataset. !!
Logistic regression does not support imbalanced classification directly. Instead, the training algorithm used to fit the logistic regression model must be modified to take the skewed distribution into account.
Imbalanced classification is specifically hard because of the severely skewed class distribution and the unequal misclassification costs. The difficulty of imbalanced classification is compounded by properties such as dataset size, label noise, and data distribution.
Why is this a problem? Most machine learning algorithms assume data equally distributed. So when we have a class imbalance, the machine learning classifier tends to be more biased towards the majority class, causing bad classification of the minority class.
Class imbalance problem become greatest issue in data mining. Imbalance problem occur where one of the two classes having more sample than other classes. The most of algorithm are more focusing on classification of major sample while ignoring or misclassifying minority sample.
Why don't you use the InfogainLoss layer to compensate for the imbalance in your training set?
The Infogain loss is defined using a weight matrix H
(in your case 2-by-2) The meaning of its entries are
[cost of predicting 1 when gt is 0, cost of predicting 0 when gt is 0 cost of predicting 1 when gt is 1, cost of predicting 0 when gt is 1]
So, you can set the entries of H
to reflect the difference between errors in predicting 0 or 1.
You can find how to define matrix H
for caffe in this thread.
Regarding sample weights, you may find this post interesting: it shows how to modify the SoftmaxWithLoss layer to take into account sample weights.
Recently, a modification to cross-entropy loss was proposed by Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, Piotr Dollár Focal Loss for Dense Object Detection, (ICCV 2017).
The idea behind focal-loss is to assign different weight for each example based on the relative difficulty of predicting this example (rather based on class size etc.). From the brief time I got to experiment with this loss, it feels superior to "InfogainLoss"
with class-size weights.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With