how does TensorFlow handle the differentials for L1 regularization?

Question

it seems that you can just declare a cost function by tf.abs() and then pass it down to auto-gradient generation (see https://github.com/nfmcclure/tensorflow_cookbook/blob/master/03_Linear_Regression/04_Loss_Functions_in_Linear_Regressions/04_lin_reg_l1_vs_l2.py)

. but we know abs() is not differentiable.

how is this done in Tensorflow? does it just randomly throw a number in [-1,1] ?

if someone could please point me to the implementation that would be great. Thanks!

(I looked for tensorflow.py in the git, but it does not even exist)

Answer 1

f(x) = abs(x) is differentiable everywhere, except at x=0. It derivative equals:

So the only question is how tensorflow implements derivative at x=0. You can check this manually:

import tensorflow as tf
x = tf.Variable(0.0)
y = tf.abs(x)
grad = tf.gradients(y, [x])[0]
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print(sess.run(grad))

It prints 0.0.