Attention Layer throwing TypeError: Permute layer does not support masking in Keras

Question

I have been following this post in order to implement attention layer over my LSTM model.

Code for the attention layer:

INPUT_DIM = 2
TIME_STEPS = 20
SINGLE_ATTENTION_VECTOR = False
APPLY_ATTENTION_BEFORE_LSTM = False

def attention_3d_block(inputs):
    input_dim = int(inputs.shape[2])
    a = Permute((2, 1))(inputs)
    a = Reshape((input_dim, TIME_STEPS))(a)
    a = Dense(TIME_STEPS, activation='softmax')(a)
    if SINGLE_ATTENTION_VECTOR:
        a = Lambda(lambda x: K.mean(x, axis=1), name='dim_reduction')(a)
        a = RepeatVector(input_dim)(a)
    a_probs = Permute((2, 1), name='attention_vec')(a)
    output_attention_mul = merge(
        [inputs, a_probs],
        name='attention_mul',
        mode='mul'
    )
    return output_attention_mul

The error I get:

File "main_copy.py", line 244, in <module>
     model = create_model(X_vocab_len, X_max_len, y_vocab_len, y_max_len, HIDDEN_DIM, LAYER_NUM)
File "main_copy.py", line 189, in create_model
     attention_mul = attention_3d_block(temp)
File "main_copy.py", line 124, in attention_3d_block
     a = Permute((2, 1))(inputs)
File "/root/.virtualenvs/keras_tf/lib/python3.5/site-packages/keras/engine/topology.py", line 597, in __call__
     output_mask = self.compute_mask(inputs, previous_mask)
File "/root/.virtualenvs/keras_tf/lib/python3.5/site-packages/keras/engine/topology.py", line 744, in compute_mask
     str(mask)) TypeError: Layer permute_1 does not support masking, but was passed an input_mask: Tensor("merge_2/All:0", shape=(?, 15),
 dtype=bool)

I went through this thread which says:

It is a small change in the Keras source code (set the supports_masking class variable in the Lambda layer to True instead of False). Otherwise there isn't a way to do this. Masking isn't really necessary though.

Where can I set the supports_masking variable to True? Also, is there any other solution to this?

Answer 1

I am one of the authors of the package.

You should use the latest version. The previous versions had some conceptual problems.