tf.contrib.layers.embed_sequence() is for what?

Question

I found tf.contrib.layers.embed_sequence() function in the lastest Tensorflow examples, but it is not included in the main API . I don't know why. Any explanation about how it works would be appreciated.

Answer 1

I can think of two main reasons why tensorflow.contrib.layers.embed_sequence is useful:

1. When building a neural network model that has multiple gates that take features as input, by using tensorflow.contrib.layers.embed_sequence, you can reduce the number of parameters in your network while preserving depth. For example, it eliminates the need for each gates of the LSTM to perform its own linear projection of features.
2. It allows for arbitrary input shapes, which helps the implementation be simple and flexible.

Let us say that I have a data set which looks something like this:

[("garbage piles in the city","Garbage"), ("city is clogged with vehicles","Traffic")]

I want to take the first element of each tuple which is a sequence of words. The words need to be embedded in a vector form. As the first step, they should be converted as indices or numbers. For example, in this case, the vocabulary will be:

vocab = [{'garbage':1},
         {'piles':2},
         {'in':3},
         {'the':4},
         {'city':5},
         {'is':6},
         {'clogged':7},
         {'with':8},
         {'vehicles':9}]

The encoded text will look like this:

features = [[1, 2, 3, 4, 5], [5, 6, 7, 8, 9]]

You pass this encoded text as features to this function in batches:

features_embedded = tf.contrib.layers.embed_sequence(
    ids=features,
    vocab_size=len(vocab),
    embed_dim=EMBEDDING_SIZE,
    scope='words'
)

Now, every word which is represented using the indices (1 to 5), becomes embedded into a vector of size EMBEDDING_SIZE.

If the batch size is 2 (ie. 2 sequences in one batch) and EMBEDDING_SIZE is 10, the output will be a matrix of shape (2, 5, 10)

Sample output:

[[[0.1, 0.3, 0.4, 0.2, 0.5, 0.2, 0.2, 0.2, 0.4, 0.1], # garbage
  [0.1, 0.3, 0.4, 0.2, 0.5, 0.2, 0.1, 0.2, 0.4, 0.1], # piles
  [0.1, 0.3, 0.4, 0.2, 0.5, 0.2, 0.4, 0.2, 0.4, 0.1], # in
  [0.1, 0.3, 0.4, 0.2, 0.5, 0.3, 0.1, 0.2, 0.4, 0.1], # the
  [0.1, 0.3, 0.4, 0.2, 0.5, 0.2, 0.1, 0.2, 0.4, 0.6]], # city
 [sent2]]

sent2 is encoded similarly ( 5 x 10 matrix).

Hope this is clear.