Generative adversarial networks tanh? [closed]

Question

I was wondering, why in most of the models of GAN (in MNIST at least) I saw, the activation function (for the discriminator and the generator) was tanh ? Isn't ReLu more efficient ? (I always read that for predictive networks)

Thanks!

Answer 1

From the DCGAN paper [Radford et al. https://arxiv.org/pdf/1511.06434.pdf]...

"The ReLU activation (Nair & Hinton, 2010) is used in the generator with the exception of the output layer which uses the Tanh function. We observed that using a bounded activation allowed the model to learn more quickly to saturate and cover the color space of the training distribution. Within the discriminator we found the leaky rectified activation (Maas et al., 2013) (Xu et al., 2015) to work well, especially for higher resolution modeling. This is in contrast to the original GAN paper, which used the maxout activation (Goodfellow et al., 2013)."

It could be that the symmetry of tanh is an advantage here, since the network should be treating darker colours and lighter colours in a symmetric way.

Answer 2

Sometimes it depends on the range that you want the activations to fall into. Whenever you hear "gates" in ML literature, you'll probably see a sigmoid, which is between 0 and 1. In this case, maybe they want activations to fall between -1 and 1, so they use tanh. This page says to use tanh, but they don't give an explanation. DCGAN uses ReLUs or leaky ReLUs except for the output of the generator. Makes sense - what if half of your embedding becomes zeros? Might be better to have a smoothly varying embedding between -1 and 1.

I'd love to hear someone else's input, as I'm not sure.