Assertion failed: predictions must be >= 0, Condition x >= y did not hold element-wise

Question

I am running a multi-class model(total 40 class in total) for 2000 epochs. The model is running fine till 828 epoch but at 829 epoch it is giving me an InvalidArgumentError (see the screenshot below)

Below is the code that I used to build my model.

n_cats = 40 
input_bow = tf.keras.Input(shape=(40), name="bow")
hidden_1 = tf.keras.layers.Dense(200, activation="relu")(input_bow)

hidden_2 = tf.keras.layers.Dense(100, activation="relu")(hidden_1)

hidden_3 = tf.keras.layers.Dense(80, activation="relu")(hidden_2)

hidden_4 = tf.keras.layers.Dense(70, activation="relu")(hidden_3)

output = tf.keras.layers.Dense(n_cats, activation="sigmoid")(hidden_4)

model = tf.keras.Model(inputs=[input_bow], outputs=output)

METRICS = [
    tf.keras.metrics.Accuracy(name="Accuracy"),
    tf.keras.metrics.Precision(name="precision"),
    tf.keras.metrics.Recall(name="recall"),
    tf.keras.metrics.AUC(name="auc"),
    tf.keras.metrics.BinaryAccuracy(name="binaryAcc")
]

checkpoint_cb = tf.keras.callbacks.ModelCheckpoint(
    "my_keras_model.h5", save_best_only=True)
lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(initial_learning_rate=1e-2,
                                                             decay_steps=10000,
                                                             decay_rate=0.9)


adam_optimizer = tf.keras.optimizers.Adam(learning_rate=lr_schedule)
model.compile(loss="categorical_crossentropy",
              optimizer="adam", metrics=METRICS)

training_history = model.fit(
    (bow_train),
    indus_cat_train,
    epochs=2000,
    batch_size=128,
    callbacks=[checkpoint_cb],
    validation_data=(bow_test, indus_cat_test))

Please help me to understand this behavior of TensorFlow. What is causing this error? I have read this and this but these do not seem to be a correct explanation in my case.

Answer 1

I think that this error is due to the setting of the AUC metric.(see https://www.tensorflow.org/api_docs/python/tf/keras/metrics/AUC) The predictions should be all non-negative values instead of [-nan, -nan, ...] as your model output. You can try something from http://deeplearning.net/software/theano/tutorial/nan_tutorial.html to deal with the NANs. And, if you want to quickly solve this error, you can directly remove the AUC metric from the list.