Keras summation Layer acting weird, summing over training set

Question

I am having trouble understanding the basic way Keras works. I am experimenting with a single summation layer, implemented as a Lambda layer using tensorflow as a backend:

from keras import backend as K

test_model = Sequential()
test_model.add( Lambda( lambda x: K.sum(x, axis=0), input_shape=(2,3)) )

x = np.reshape(np.arange(12), (2,2,3))
test_model.predict(x)

This returns:

array([[  6.,   8.,  10.],
   [ 12.,  14.,  16.]], dtype=float32)

Which is very weird, as it sums over the first index, which to my understanding corresponds to the index of the training data. Also, if I change the axis to axis=1 then the sum is taken over the second coordinate, which is what I would expect to get for axis=0.

What is going on? Why does it seem like the axis chosen effects how the data is passed to the lambda layer?

Answer 1

The input_shape is the shape of one sample of the batch.
It doesn't matter if you have 200 or 10000 samples in a batch, all the samples should be (2,3).

But the batch itself is what is passed along from one layer to another.
A batch contains "n" samples, each sample with the input_shape:

• Batch shape then is: (n, 2, 3) -- n samples, each sample with input_shape = (2,3)

You don't define "n" when input_shape is required, because "n" will be defined when you use fit or another training command, with the batch_size. (In your example, n = 2)

---

This is the original array:

[[[ 0  1  2]
  [ 3  4  5]]

 [[ 6  7  8]
  [ 9 10 11]]]

Sample 1 = [ 0  1  2], [ 3  4  5]
Sample 2 = [ 6  7  8], [ 9 10 11]

Summing on index 0 (the batch size dimension) will sum sample 1 with sample 2:

[ 6 8 10], [12 14 16]

Summing on index 1 will sum the first dimension of one sample's input shape:

[ 3, 5, 7 ], [15, 17, 19]