I am using google-colaboratory to train some convnets on image classification with Keras. And I get often disconnected from the notebook which I've found (on some forums) to be a frequent issue with some colaboratory users. And when I try to reconnect it gets stuck on "initializing":
So I decided to use Keras Checkpoint to save the model I'm training on my google drive, after each epoch.
During the training the notebook disconnected after 2 epochs, but when I monitored my google drive I've observed that models are being saved after each epoch, despite my colaboratory being disconnected.
My guess is that programs continue their execution in the background even if my noteboook is disconnected.
My question is, Am I guessing it right? And what do these disconnections really mean? (Are our variables, etc., preserved)?
In fact, if we leave the notebook idle for more than 30 minutes, Google Colab automatically disconnects it. Every 60 seconds, this function clicks the connect button. As a result, Colab believes that the notebook is not idle, and you should not be concerned about being disconnected!
Once you are in your Colab notebook you can use it just like any offline Jupyter notebook.
The current cell will continue executing once you close your browser, but the outputs will not end up in the notebook in Drive.
After mounting Google Drive, you can download datasets into Google Drive and use it in Colab session, or save outputs of the session into your Google Drive.
Normally, notebook connection expires after 12 hours and all the data loaded into colab will also be expired with session.
When you run a process and the notebook get disconnected after some time but before 12 hours, the code run in background.
So, if you try to reconnect the notebook, it get stuck on "initializing" until the completion of code execution or session expiry (whichever occurs first). After that, the connection is established again.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With