When to stop training neural networks?

Question

I'm trying to carry out a domain-specific classification research using RNN and have accumulated tens of millions of texts. Since it takes days and even months to run the whole dataset over, I only picked a small portion of it for testing, say 1M texts (80% for training, 20% for validation). I pre-trained the whole corpus with word vectorization and I also applied Dropout to the model to avoid over-fitting. When it trained 60000 text within 12 hrs, the loss had already dropped to a fairly low level with the accuracy 97%. Should I continue or not? Does it help continue with the training?

It is still running the first epoch and I'm afraid if I stopped right now, the model wouldn't cover the whole...

Answer 1

Well as a general rule of thumb you may want let the model stay in training phase until the validation starts dropping for several consecutive iterations. After that point the model has strated to over-fit the data. For your problem, since it takes a very long time to train the model I suggest you stop the training after the first epoch and test the model and make sure there is no implementational bugs in your code. If the model has an acceptable accuracy then start training it again.