Why is input_length needed in layers.Embedding in keras tensorflow?

Question

layers.embedding has a parameter (input_length) that the documentation describes as:

input_length : Length of input sequences, when it is constant. This argument is required if you are going to connect Flatten then Dense layers upstream (without it, the shape of the dense outputs cannot be computed).

Why is it that the shape of dense outputs cannot be computed. To me, Flatten seems pretty easy to do. It is just a tf.rehshape(input,(-1,1)) followed by a dense layer with whatever output shape we choose.

Can you help me point to the lapse in my understanding of the entir logic?

Answer 1

By specifying the dimension, you're making sure the model receives fixed-length input.

Technically, you can just put None at any input dimension you want. The shape will be inferred at run-time.

You only need to make sure you're specifying the layer parameters (input_dim, output_dim), kernel_size (for conv layers), units (for FC layers).

The shape can be computed if you use Input and specify what shape of tensor will be passed through the network.

For example following model is perfectly valid:

from tensorflow.keras import layers
from tensorflow.keras import models

ip = layers.Input((10))
emb = layers.Embedding(10, 2)(ip)
flat = layers.Flatten()(emb)
out = layers.Dense(5)(flat)

model = models.Model(ip, out)

model.summary()

Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_2 (InputLayer)         [(None, 10)]              0         
_________________________________________________________________
embedding (Embedding)        (None, 10, 2)             20        
_________________________________________________________________
flatten (Flatten)            (None, 20)                0         
_________________________________________________________________
dense (Dense)                (None, 5)                 105       
=================================================================
Total params: 125
Trainable params: 125
Non-trainable params: 0

Here, I didn't specify the input_length but it was inferred from the Input layer.

The problem is with Sequential API, if you don't specify the input shape in the Input layer and also not in the embedding layer, there's no way the model can be built with the proper set of parameters.

For example,

from tensorflow.keras import layers
from tensorflow.keras import models

model = models.Sequential()
model.add(layers.Embedding(10, 2, input_length = 10)) # will be an error if I don't specify input_length here as there is no way to know the shape of the next layers without knowing the length

model.add(layers.Flatten())
model.add(layers.Dense(5))


model.summary()

In this example, you must specify the input_length, otherwise the model will throw error.