Difference between Tensorflow and Scikitlearn log_loss function implementation

Question

Hi I am trying to get into tensorflow and feeling a bit dumb. Does log_loss in TF differ from sklearn's one?

Here are some lines from my code, how I am calculating:

from sklearn.metrics import log_loss

tmp = np.array(y_test)
y_test_t = np.array([tmp, -(tmp-1)]).T[0]

tf_log_loss = tf.losses.log_loss(predictions=tf.nn.softmax(logits), labels=tf_y)

with tf.Session() as sess:

    # training

    a = sess.run(tf.nn.softmax(logits), feed_dict={tf_x: xtest, keep_prob: 1.})
    print("    sk.log_loss: ", log_loss(y_test, a,eps=1e-7 ))
    print("    tf.log_loss: ", sess.run(tf_log_loss, feed_dict={tf_x: xtest, tf_y: y_test_t, keep_prob: 1.}))

Output I get

Epoch  7, Loss:     0.4875 Validation Accuracy: 0.818981
    sk.log_loss:  1.76533018874
    tf.log_loss:  0.396557
Epoch  8, Loss:     0.4850 Validation Accuracy: 0.820738
    sk.log_loss:  1.77217639627
    tf.log_loss:  0.393351
Epoch  9, Loss:     0.4835 Validation Accuracy: 0.823374
    sk.log_loss:  1.78479079656
    tf.log_loss:  0.390572

Seems like while tf.log_loss converges sk.log_loss diverges.

Answer 1

I had the same problem. After looking up the source code of tf.losses.log_loss, its key lines show wat is going on:

losses = - math_ops.multiply(labels, math_ops.log(predictions + epsilon))
    - math_ops.multiply((1 - labels), math_ops.log(1 - predictions + epsilon))

It is binary log-loss (i.e. every class is considered non-exclusive) rather than multi-class log-loss.

As I worked with probabilities (rather than logits), I couldn't use tf.nn.softmax_cross_entropy_with_logits (though, I could have applied logarithm). My solution was to implement log-loss by hand:

loss = tf.reduce_sum(tf.multiply(- labels, tf.log(probs))) / len(probs)

See also:

• https://github.com/tensorflow/tensorflow/issues/2462
• difference between tensorflow tf.nn.softmax and tf.nn.softmax_cross_entropy_with_logits