Tensorboard + Keras + ML Engine

Question

I currently have Google Cloud ML Engine setup to train models created in Keras. When using Keras, it seems ML Engine does not automatically save the logs to a storage bucket. I see the logs in the ML Engine Jobs page but they do not show in my storage bucket and therefore I am unable to run tensorboard while training.

You can see the job completed successfully and produced logs:

But then there are no logs saved in my storage bucket:

I followed this tutorial when setting up my environment: (http://liufuyang.github.io/2017/04/02/just-another-tensorflow-beginner-guide-4.html)

So, how do I get the logs and run tensorboard when training a Keras model on ML Engine? Has anyone else had success with this?

Answer 1

You will need to create a callback keras.callbacks.TensorBoard(..) in order to write out the logs. See Tensorboad callback. You can supply GCS path as well (gs://path/to/my/logs) to the log_dir argument of the callback and then point Tensorboard to that location. You will add the callback as a list when calling model.fit_generator(...) or model.fit(...).

tb_logs = callbacks.TensorBoard(
            log_dir='gs://path/to/logs',
            histogram_freq=0,
            write_graph=True,
            embeddings_freq=0)

model.fit_generator(..., callbacks=[tb_logs])