Merging layers on Keras (dot product)

Question

I've been following Towards Data Science's tutorial about word2vec and skip-gram models, but I stumbled upon a problem that I cannot solve, despite searching about it for hours and trying a lot of unsuccessful solutions.

https://towardsdatascience.com/understanding-feature-engineering-part-4-deep-learning-methods-for-text-data-96c44370bbfa

The step that it shows you how to build the skip-gram model architecture seems deprecated because of the use of the Merge layer from keras.layers.

I've seem many discussions about it, and the majority of answers was the you need to use the Functional API of Keras to merge layers now. But the problem is, I'm a total beginner in Keras and have no idea how to translate my code from Sequential to Functional, here's the code that the author used (and I copied):

from keras.layers import Merge
from keras.layers.core import Dense, Reshape
from keras.layers.embeddings import Embedding
from keras.models import Sequential

# build skip-gram architecture
word_model = Sequential()
word_model.add(Embedding(vocab_size, embed_size,
                         embeddings_initializer="glorot_uniform",
                         input_length=1))
word_model.add(Reshape((embed_size, )))

context_model = Sequential()
context_model.add(Embedding(vocab_size, embed_size,
                  embeddings_initializer="glorot_uniform",
                  input_length=1))
context_model.add(Reshape((embed_size,)))

model = Sequential()
model.add(Merge([word_model, context_model], mode="dot"))
model.add(Dense(1, kernel_initializer="glorot_uniform", activation="sigmoid"))
model.compile(loss="mean_squared_error", optimizer="rmsprop")

# view model summary
print(model.summary())

# visualize model structure
from IPython.display import SVG
from keras.utils.vis_utils import model_to_dot

SVG(model_to_dot(model, show_shapes=True, show_layer_names=False, 
rankdir='TB').create(prog='dot', format='svg'))

And when I run the block, the following error is shown:

ImportError                               Traceback (most recent call last)
<ipython-input-79-80d604373468> in <module>()
----> 1 from keras.layers import Merge
      2 from keras.layers.core import Dense, Reshape
      3 from keras.layers.embeddings import Embedding
      4 from keras.models import Sequential
      5 

ImportError: cannot import name 'Merge'

What I'm asking here is some guidance on how to transform this Sequential into a Functional API structure.

Answer 1

This did indeed change. For a dot product, you can now use the dot layer:

from keras.layers import dot
...
dot_product = dot([target, context], axes=1, normalize=False)
...

You have to set the axis parameter according to your data, of course. If you set normalize=True, this gives the cosine proximity. For more information, see the documentation.

To learn about the functional API to Keras, there is a good guide to the functional API in the documentation. It's not difficult to switch if you already understand the sequential API.

Answer 2

Merge seems deprecated so Instead of Merge use Dot directly on embedding (and not with models). Use below code.

from keras.layers import Input
from keras.models import Model
from keras.layers.embeddings import Embedding
from keras.layers.core import Dense, Reshape
from keras.layers import dot

input_target = Input((1,))
input_context = Input((1,))

embedding = Embedding(vocab_size, embed_size, input_length=1, name='embedding')

word_embedding = embedding(input_target)
word_embedding = Reshape((embed_size, 1))(word_embedding)
context_embedding = embedding(input_context)
context_embedding = Reshape((embed_size, 1))(context_embedding)

# now perform the dot product operation  
dot_product = dot([word_embedding, context_embedding], axes=1)
dot_product = Reshape((1,))(dot_product)

# add the sigmoid output layer
output = Dense(1, activation='sigmoid')(dot_product)

model = Model(input=[input_target, input_context], output=output)
model.compile(loss='mean_squared_error', optimizer='rmsprop')

# view model summary
print(model.summary())

# visualize model structure
from IPython.display import SVG
from keras.utils.vis_utils import model_to_dot

SVG(model_to_dot(model, show_shapes=True, show_layer_names=False, 
                 rankdir='TB').create(prog='dot', format='svg'))