keras usage of the Activation layer instead of activation parameter

Question

There is an Activation layer in Keras.

Seems this code:

  model.add(Convolution2D(64, 3, 3))
  model.add(Activation('relu'))

and this one:

  model.add(Convolution2D(64, 3, 3, activation='relu'))

produces the same result.

What is the purpose of this additional Activation layer?

[Upgr: 2017-04-10] Is there a difference in performance with above two scenarios?

Answer 1

As you may see, both approaches are equivalent. I will show you a few scenarios in which having this layer might help:

1. Same layer - different activations- one may easily imagine a net where you want to have different activations applied to the same output. Without Activation it's impossible.
2. Need for output before activation - e.g. in siamese networks you are training your network using softmax as final activation - but in the end - you want to have so called logits - inverse softmax function. Without additional Activation layer that could be difficult.
3. Saliency maps: in saliency maps - similiar to what you have in a previous point - you also need output before activation in order to compute a gradient w.r.t. to it - without Activation it wouldn't be possible.

As you may see - lack of Activation would make output of a layer before activation and final activation strongly coupled. That's why Activation might be pretty useful - as it breaks this ties.