ValueError: dimension of the inputs to `Dense` should be defined. Found `None`

Question

I've been working on a TensorFlow 2 model, but I am constantly hitting this error. I have tried to define the shape for every layer and still no change. Furthermore, the error only shows up when I specify sparse=True in the input layer, which I have to specify as my input tensors are sparse and other parts of the script require it. Tensorflow version: Version: 2.0.0-beta1. If I use a newer version than this, other obscure errors appear because of the sparse input. It's remarkable how many issues TF 2.0 seems to have with this type of input.

The current method definition:

def make_feed_forward_model():
    #'''
    inputs = tf.keras.Input(shape=(HPARAMS.max_seq_length,),dtype='float32', name='sample', sparse=True)
    dense_layer_1 = tf.keras.layers.Dense(HPARAMS.num_fc_units, activation='relu')(inputs)
    dense_layer_2 = tf.keras.layers.Dense(HPARAMS.num_fc_units_2, activation='relu')(dense_layer_1)
    dense_layer_3 = tf.keras.layers.Dense(HPARAMS.num_fc_units_3, activation='relu')(dense_layer_2)
    outputs = tf.keras.layers.Dense(4, activation='softmax')(dense_layer_3)

    return tf.keras.Model(inputs=inputs, outputs=outputs)
    #'''

Then when I run the following, the error appears:

model = make_feed_forward_model()
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

Traceback:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-56-720f117bb231> in <module>
      1 # Feel free to use an architecture of your choice.
----> 2 model = make_feed_forward_model()
      3 model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

<ipython-input-55-5f35f6f22300> in make_feed_forward_model()
     18     #embedding_layer = tf.keras.layers.Embedding(HPARAMS.vocab_size, 16)(inputs)
     19     #pooling_layer = tf.keras.layers.GlobalAveragePooling1D()(inputs)
---> 20     dense_layer_1 = tf.keras.layers.Dense(HPARAMS.num_fc_units, activation='relu')(inputs)
     21     dense_layer_2 = tf.keras.layers.Dense(HPARAMS.num_fc_units_2, activation='relu')(dense_layer_1)
     22     dense_layer_3 = tf.keras.layers.Dense(HPARAMS.num_fc_units_3, activation='relu')(dense_layer_2)

~\Anaconda3\envs\tf-nsl\lib\site-packages\tensorflow\python\keras\engine\base_layer.py in __call__(self, inputs, *args, **kwargs)
    614           # Build layer if applicable (if the `build` method has been
    615           # overridden).
--> 616           self._maybe_build(inputs)
    617 
    618           # Wrapping `call` function in autograph to allow for dynamic control

~\Anaconda3\envs\tf-nsl\lib\site-packages\tensorflow\python\keras\engine\base_layer.py in _maybe_build(self, inputs)
   1964         # operations.
   1965         with tf_utils.maybe_init_scope(self):
-> 1966           self.build(input_shapes)
   1967       # We must set self.built since user defined build functions are not
   1968       # constrained to set self.built.

~\Anaconda3\envs\tf-nsl\lib\site-packages\tensorflow\python\keras\layers\core.py in build(self, input_shape)
   1003     input_shape = tensor_shape.TensorShape(input_shape)
   1004     if tensor_shape.dimension_value(input_shape[-1]) is None:
-> 1005       raise ValueError('The last dimension of the inputs to `Dense` '
   1006                        'should be defined. Found `None`.')
   1007     last_dim = tensor_shape.dimension_value(input_shape[-1])

ValueError: The last dimension of the inputs to `Dense` should be defined. Found `None`.

EDIT: SparseTensor errors

It seems that if I use any version newer than TF 2.0.0-beta1, training completely fails:

ValueError: The two structures don't have the same nested structure.

    First structure: type=TensorSpec str=TensorSpec(shape=(None, None), dtype=tf.float32, name=None)

    Second structure: type=SparseTensor str=SparseTensor(indices=Tensor("sample/indices_1:0", shape=(None, 2), dtype=int64), values=Tensor("sample/values_1:0", shape=(None,), dtype=float32), dense_shape=Tensor("sample/shape_1:0", shape=(2,), dtype=int64))

    More specifically: Substructure "type=SparseTensor str=SparseTensor(indices=Tensor("sample/indices_1:0", shape=(None, 2), dtype=int64), values=Tensor("sample/values_1:0", shape=(None,), dtype=float32), dense_shape=Tensor("sample/shape_1:0", shape=(2,), dtype=int64))" is a sequence, while substructure "type=TensorSpec str=TensorSpec(shape=(None, None), dtype=tf.float32, name=None)" is not
    Entire first structure:
    .
    Entire second structure:
    .

EDIT 2: Error after adding batch_size to the Input layer:

def make_feed_forward_model():  
    inputs = tf.keras.Input(shape=(HPARAMS.max_seq_length,),dtype='float32', name='sample', sparse=True, batch_size=HPARAMS.batch_size)
    dense_layer_1 = tf.keras.layers.Dense(HPARAMS.num_fc_units, activation='relu')(inputs)
    dense_layer_2 = tf.keras.layers.Dense(HPARAMS.num_fc_units_2, activation='relu')(dense_layer_1)
    dense_layer_3 = tf.keras.layers.Dense(HPARAMS.num_fc_units_3, activation='relu')(dense_layer_2)
    outputs = tf.keras.layers.Dense(4, activation='softmax')(dense_layer_3)

    return tf.keras.Model(inputs=inputs, outputs=outputs)

model = make_feed_forward_model()
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

When I run model.compile():

TypeError: Failed to convert object of type <class 'tensorflow.python.framework.sparse_tensor.SparseTensor'> to Tensor. 

Contents: SparseTensor(indices=Tensor("sample/indices_3:0", shape=(None, 2), dtype=int64), values=Tensor("sample/values_3:0", shape=(None,), dtype=float32), dense_shape=Tensor("sample/shape_3:0", shape=(2,), dtype=int64)). Consider casting elements to a supported type.

Answer 1

This happens because when input tensor is sparse shape of this tensor evaluates to (None,None) instead of (HPARAMS.max_seq_length,)

inputs = tf.keras.Input(shape=(100,),dtype='float32', name='sample', sparse=True)
print(inputs.shape)
# output: (?, ?)

This also seems to be an open issue.
One solution is to write custom layer sub-classing Layer class (Refer this).

As a work-around (tested on tf-gpu 2.0.0) adding batch-size in input layer works fine:

inputs = tf.keras.Input(shape=(100,),dtype='float32', name='sample', sparse=True ,batch_size=32)
print(inputs.shape)
# output: (32, 100)