TensorBoard training plot has less steps than the validation one. Why?

Question

I am training a neural network using TensorFlow and I want to visualize the training result with TensorBoard.

My code is the following:

model = Sequential([
    Dense(len(test_inputs[0])),
    BatchNormalization(),
    Activation('tanh'),
    Dropout(0.01),

    Dense(128),
    BatchNormalization(),
    Activation('tanh'),
    Dropout(0.01),

    Dense(128),
    BatchNormalization(),
    Activation('relu'),
    Dropout(0.01),

    Dense(len(test_outputs[0])),
    BatchNormalization(),
    Activation('softmax')
])

model.compile(
    optimizer='Adadelta',
    loss='mse',
    metrics=['accuracy']
)


log_dir="logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)


model.fit(
    x = train_inputs,
    y = train_outputs,
    epochs = 5000,
    batch_size = 100,
    validation_data = (test_inputs, test_outputs),
    callbacks = [tensorboard_callback],
    verbose = False
)

All works as expected except that the plot I get (picture below) in TensorBoard shows all the steps for the validation data (blue line) and only a few steps for the training data (red line).

Why so? Surely I am doing something wrong but I cannot figure out what.

Answer 1

This is due to the profiling used in TensorBoard. The issue is actually ongoing here.

You can fix that by either restarting the tensorboard process after the training, or if you need to be able to follow the training metrics, by preventing the profiling in the TensorBoard callback profile_batch=0.