Why does Tensorboard not refresh when used with rsync?

Question

I am running a tensorflow experiment on a remote machine continuously writing to the same events.out.tfevents.xxx file. I would expect tensorboard to refresh automatically every minute or so displaying the new logs. This does work when using sshfs to mount the remote machine on my laptop and using the mounted directory to run tensorboard on.

However, when using rsync to copy the files over and run tensorboard on the local files, the tensorboard never refreshes, I have to restart it in order to get the updates.

This is my rsync command:

rsync -aP --del -e ssh server_name:folder_on_server local_folder --exclude='*checkpoints*' --exclude='*.json' --exclude='*.DS_Store'

Any help would be greatly appreciated!

Answer 1

It's a known issue with the Tensorboard, see this issue on github.

Here's an quote from the issue (emphasis is mine) :

It looks like when the tensorboard reads an event file from local directory - it will not notice that the event file was deleted and recreated (which is quite valid case when you are using [...] rsync to sync the data)

One workaround is to use --inplace as an option in your rsync command.