as_list() is not defined on an unknown TensorShape on y_t_rank = len(y_t.shape.as_list()) and related to metrics

Question

TF 2.3.0.dev20200620

I got this error during .fit(...) for a model with a sigmoid binary output. I used tf.data.Dataset as the input pipeline. The strange thing is it depends on the metric:

Don't work:

model.compile(
    optimizer=tf.keras.optimizers.Adam(lr=1e-4, decay=1e-6),
    loss=tf.keras.losses.BinaryCrossentropy(),
    metrics=['accuracy']
)

work:

model.compile(
    optimizer=tf.keras.optimizers.Adam(lr=1e-4, decay=1e-6),
    loss=tf.keras.losses.BinaryCrossentropy(),
    metrics=[tf.keras.metrics.BinaryAccuracy()]
)

But as I understood, 'accuracy' should be fine. In fact, instead of using my own tf.data.Dataset custom setup (can be provided if needed), using tf.keras.preprocessing.image_dataset_from_directory give no such error. This is the case from tutorial https://keras.io/examples/vision/image_classification_from_scratch.

Trace is pasted below. Notice this is diff from other 2 older questions. it involves somehow the metrics.

ValueError: in user code:

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:806 train_function  *
    return step_function(self, iterator)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:796 step_function  **
    outputs = model.distribute_strategy.run(run_step, args=(data,))
/usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:1211 run
    return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2526 call_for_each_replica
    return self._call_for_each_replica(fn, args, kwargs)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2886 _call_for_each_replica
    return fn(*args, **kwargs)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:789 run_step  **
    outputs = model.train_step(data)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:759 train_step
    self.compiled_metrics.update_state(y, y_pred, sample_weight)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/compile_utils.py:388 update_state
    self.build(y_pred, y_true)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/compile_utils.py:319 build
    self._metrics, y_true, y_pred)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/nest.py:1139 map_structure_up_to
    **kwargs)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/nest.py:1235 map_structure_with_tuple_paths_up_to
    *flat_value_lists)]
/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/nest.py:1234 <listcomp>
    results = [func(*args, **kwargs) for args in zip(flat_path_list,
/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/nest.py:1137 <lambda>
    lambda _, *values: func(*values),  # Discards the path arg.
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/compile_utils.py:419 _get_metric_objects
    return [self._get_metric_object(m, y_t, y_p) for m in metrics]
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/compile_utils.py:419 <listcomp>
    return [self._get_metric_object(m, y_t, y_p) for m in metrics]
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/compile_utils.py:440 _get_metric_object
    y_t_rank = len(y_t.shape.as_list())
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/tensor_shape.py:1190 as_list
    raise ValueError("as_list() is not defined on an unknown TensorShape.")

ValueError: as_list() is not defined on an unknown TensorShape.

Answer 1

Had exactly the same problem when using 'accuracy' metric.

I followed https://github.com/tensorflow/tensorflow/issues/32912#issuecomment-550363802 example:

def _fixup_shape(images, labels, weights):
    images.set_shape([None, None, None, 3])
    labels.set_shape([None, 19]) # I have 19 classes
    weights.set_shape([None])
    return images, labels, weights
dataset = dataset.map(_fixup_shape)

which helped me solve the problem.

But, in my case, instead of using one map function, as kawingkelvin did above, to load and set_shape inside, I needed to use two map functions because of some errors in the TF code.

The final solution for me was to use the following order: dataset.batch.map(get_data).map(fix_shape).prefetch

NOTE: batch can be done both before and after map(get_data) depending on how your get_data function is created. Fix_shape must be done after.