TensorBoard 2.0.0 not updating train scalars

Question

I am running the following in a docker container:

• tensorflow-gpu 2.0.0
• tensorboard 2.0.0
• jupyterlab 1.1.4

Unfortunately, during the training Tensorboard is only updating the validation scalars graph (accuracy and loss), but not those of the training data. The directories for train and validation are both selected to be displayed within the graph.

I found out, that when I stop Tensorboard and restart it, the train scalars currently available until this time step get displayed. Unfortunately, these do not get updated, although the validation scalars do get updated with each epocch.

Does someone know the solution to this problem?

Answer 1

It's probably related to https://github.com/tensorflow/tensorboard/issues/2412.

Adding profile_batch=0 to the callback should solve the problem.

Answer 2

I had the exact same issue and solved it by starting my tensorboard with :

--reload_multifile True