How to penalize False Negatives more than False Positives

Question

From the business perspective, false negatives lead to about tenfold higher costs (real money) than false positives. Given my standard binary classification models (logit, random forest, etc.), how can I incorporate this into my model?

Do I have to change (weight) the loss function in favor of the 'preferred' error (FP) ? If so, how to do that?

Answer 1

There are several options for you:

• As suggested in the comments, class_weight should boost the loss function towards the preferred class. This option is supported by various estimators, including sklearn.linear_model.LogisticRegression, sklearn.svm.SVC, sklearn.ensemble.RandomForestClassifier, and others. Note there's no theoretical limit to the weight ratio, so even if 1 to 100 isn't strong enough for you, you can go on with 1 to 500, etc.

• You can also select the decision threshold very low during the cross-validation to pick the model that gives highest recall (though possibly low precision). The recall close to 1.0 effectively means false_negatives close to 0.0, which is what to want. For that, use sklearn.model_selection.cross_val_predict and sklearn.metrics.precision_recall_curve functions:

  y_scores = cross_val_predict(classifier, x_train, y_train, cv=3,
                               method="decision_function")
  
  precisions, recalls, thresholds = precision_recall_curve(y_train, y_scores)
  

  If you plot the precisions and recalls against the thresholds, you should see the picture like this:

  

  After picking the best threshold, you can use the raw scores from classifier.decision_function() method for your final classification.

Finally, try not to over-optimize your classifier, because you can easily end up with a trivial const classifier (which is obviously never wrong, but is useless).