Bias weights for embedding layers in Keras

Question

I am working on a feed forward NN and I am using keras embeddings. I would like to set bias weight for the embeddings, but I am not sure how to do that.

Keras Dense layers allow to specify use_bias = True and then set the bias weights. Is there an equivalent approach for Embedding layers?

Answer 1

you can use another embedding with vector length equals 1 as the bias. For example, the code below gets the embeddings and biases for input a and b, takes the dot product of two vectors, then add the biases with the dot product.

from keras.models import Model
from keras.layers import Embedding, Input, Add, Dot

a = Input(shape=(1,))
b = Input(shape=(1,))

emb_a = Embedding(num_words+1, 50)(a)
bias_a = Embedding(num_words+1, 1)(a)
emb_b = Embedding(num_words+1, 50)(b)
bias_b = Embedding(num_words+1, 1)(b)

dot = Dot(axes=-1)([a,b])
add = Add()([dot,bias_a,bias_b])