Regularization in TensorFlow using tf.contrib.layers.fully_connected

Question

fully_connected has a named parameter called weights_regularizer which accepts the regularizers provided by tf.contrib.layers, however it is not clear the trainers are able to find these regularization loss tensors or if we have to manually add them to the total loss.

Could somebody provide an example of regularization using fully_connected?

Answer 1

The regularizer outputs are added to a collection called REGULARIZATION_LOSSES (the layer calls model_variable with the regularizer).

You have to add the content of REGULARIZATION_LOSSES to your loss yourself. To get all the tensors in the REGULARIZATION_LOSSES collection, use tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES).