Buffered data was truncated after reaching the output size limit

Question

When I use Colaboratory to run my NIN model, it occurs an error in the output of training process which tells "Buffered data was truncated after reaching the output size limit." in the 61th epoch. I have no idea about this crash.Is my code still running ? How can I solve this problem?

Here is some information about the output of my training process:

Epoch 57/200 391/391 [==============================] - 53s 135ms/step - loss: 0.8365 - acc: 0.7784 - val_loss: 0.9250 - val_acc: 0.7625 Epoch 58/200  28/391 [=>............................] - ETA: 46s - loss: 0.8356 - acc: 0.7835391/391 [==============================] - 53s 136ms/step - loss: 0.8288 - acc: 0.7811 - val_loss: 0.8977 - val_acc: 0.7608 Epoch 59/200 326/391 [========================>.....] - ETA: 8s - loss: 0.8309 - acc: 0.7789391/391 [==============================] - 53s 136ms/step - loss: 0.8297 - acc: 0.7798 - val_loss: 0.9030 - val_acc: 0.7628 Epoch 60/200 391/391 [==============================] - 53s 134ms/step - loss: 0.8245 - acc: 0.7825 - val_loss: 0.8378 - val_acc: 0.7767 Epoch 61/200  28/391 [=>............................] - ETA: 46s - loss: 0.8281 - acc: 0.7879390/391 [============================>.] - ETA: 0s - loss: 0.8177 - acc: 0.7851Buffered data was truncated after reaching the output size limit. 

Answer 1

Even if RAM | GPU | DISK on colab is free, this error still comes because there is a limited memory for displaying output of a cell on colab. Assuming the memory limit is around 2Mb to 5Mb when we run many epochs(148+) during training, it tends to fill that memory and hence the output is truncated because there is no more memory left free to display the buffered epochs. However, the machine keeps running in the background and the output is processed but it is not displayed because of the buffered limit. You will still get your desired output.

One solution is not to use verbose=1 (use 0 instead).