TypeError: Layer input_spec must be an instance of InputSpec. Got: InputSpec(shape=(None, 128, 768), ndim=3)

Question

I am trying to use a BERT pretrained model to do a multiclass classification (of 3 classes). Here's my function to use the model and also added some extra functionalities:

def create_model(max_seq_len, bert_ckpt_file):

  with tf.io.gfile.GFile(bert_config_file, "r") as reader:
      bc = StockBertConfig.from_json_string(reader.read())
      bert_params = map_stock_config_to_params(bc)
      bert_params.adapter_size = None
      bert = BertModelLayer.from_params(bert_params, name="bert")
        
  input_ids = keras.layers.Input(shape=(max_seq_len, ), dtype='int32', name="input_ids")
  bert_output = bert(input_ids)

  print("bert shape", bert_output.shape)

  cls_out = keras.layers.Lambda(lambda seq: seq[:, 0, :])(bert_output)
  cls_out = keras.layers.Dropout(0.5)(cls_out)
  logits = keras.layers.Dense(units=768, activation="tanh")(cls_out)
  logits = keras.layers.Dropout(0.5)(logits)
  logits = keras.layers.Dense(units=len(classes), activation="softmax")(logits)

  model = keras.Model(inputs=input_ids, outputs=logits)
  model.build(input_shape=(None, max_seq_len))

  load_stock_weights(bert, bert_ckpt_file)
        
  return model

Now when I am trying to call the function, I am getting error. The parameter values have max_seq_len = 128, bert_ckpt_file = bert checkpoint file.

model = create_model(data.max_seq_len, bert_ckpt_file)

I am getting the following error:

TypeError                                 Traceback (most recent call last)
<ipython-input-41-9609c396a3ce> in <module>()
----> 1 model = create_model(data.max_seq_len, bert_ckpt_file)

5 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/autograph/impl/api.py in wrapper(*args, **kwargs)
    693       except Exception as e:  # pylint:disable=broad-except
    694         if hasattr(e, 'ag_error_metadata'):
--> 695           raise e.ag_error_metadata.to_exception(e)
    696         else:
    697           raise

TypeError: in user code:

    /usr/local/lib/python3.7/dist-packages/bert/model.py:80 call  *
        output           = self.encoders_layer(embedding_output, mask=mask, training=training)
    /usr/local/lib/python3.7/dist-packages/keras/engine/base_layer.py:1030 __call__  **
        self._maybe_build(inputs)
    /usr/local/lib/python3.7/dist-packages/keras/engine/base_layer.py:2659 _maybe_build
        self.build(input_shapes)  # pylint:disable=not-callable
    /usr/local/lib/python3.7/dist-packages/bert/transformer.py:209 build
        self.input_spec = keras.layers.InputSpec(shape=input_shape)
    /usr/local/lib/python3.7/dist-packages/keras/engine/base_layer.py:2777 __setattr__
        super(tf.__internal__.tracking.AutoTrackable, self).__setattr__(name, value)  # pylint: disable=bad-super-call
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/training/tracking/base.py:530 _method_wrapper
        result = method(self, *args, **kwargs)
    /usr/local/lib/python3.7/dist-packages/keras/engine/base_layer.py:1297 input_spec
        'Got: {}'.format(v))

    TypeError: Layer input_spec must be an instance of InputSpec. Got: InputSpec(shape=(None, 128, 768), ndim=3)

Answer 1

For that to work you just need to downgrade tensorflow to 2.0.0 as follows :