record the computation time for each epoch in Keras during model.fit()

Question

I want to compare the computation time between different models. During the fit the computation time per epoch is printed to the console.

Epoch 5/5 160000/160000 [==============================] - **10s** ...... 

I'm looking for a way to store these times in a similar way to the model metrics that are saved in each epoch and avaliable through the history object.

Answer 1

Try the following callback:

class TimeHistory(keras.callbacks.Callback):     def on_train_begin(self, logs={}):         self.times = []      def on_epoch_begin(self, batch, logs={}):         self.epoch_time_start = time.time()      def on_epoch_end(self, batch, logs={}):         self.times.append(time.time() - self.epoch_time_start) 

Then:

time_callback = TimeHistory() model.fit(..., callbacks=[..., time_callback],...) times = time_callback.times 

In this case times should store the epoch computation times.

Answer 2

refer to answers of Marcin Możejko

import time  class TimeHistory(keras.callbacks.Callback):     def on_train_begin(self, logs={}):         self.times = []      def on_epoch_begin(self, epoch, logs={}):         self.epoch_time_start = time.time()      def on_epoch_end(self, epoch, logs={}):         self.times.append(time.time() - self.epoch_time_start) 

then

time_callback = TimeHistory() model.fit(..., callbacks=[..., time_callback],...) 

excution log

Train on 17000 samples, validate on 8000 samples Epoch 1/3 17000/17000 [==============================] - 5s 266us/step - loss: 36.7562 - mean_absolute_error: 4.5074 - val_loss: 34.2384 - val_mean_absolute_error: 4.3929 Epoch 2/3 17000/17000 [==============================] - 4s 253us/step - loss: 33.5529 - mean_absolute_error: 4.2956 - val_loss: 32.0291 - val_mean_absolute_error: 4.2484 Epoch 3/3 17000/17000 [==============================] - 5s 265us/step - loss: 31.0547 - mean_absolute_error: 4.1340 - val_loss: 30.6292 - val_mean_absolute_error: 4.1480 

then

print(time_callback.times) 

output

[4.531331300735474, 4.308278322219849, 4.505300283432007]