machine learning preventing overfitting is this cheating?

Question

When evaluating train/test results during training say these values come:

Train: 50.5% - Test: 50.2%
Train: 55.5% - Test: 53.2%
Train: 62.5% - Test: 58.2%
Train: 69.5% - Test: 62.2%
Train: 75.5% - Test: 59.2% <--- stop training, overfitting seems to be happening.

Can you stop a model from training when this occurs, or is it cheating? I.e stopping the model from training when the test accuracy decreases. I know that you are supposed to only evaluate the training set during training so that is why I am wondering.

Answer 1

This is cheating. Model trained in such a way does not have any reasonable guarantees about how it will perform "in real life"

What you are supposed to do is to have a separate mechanism of detecting when learning is finished (it seems that your training is iterative), which might either analyze the model itself, or use some separate (often called Validation) hold-out part of the dataset.

In short: you cannot use test data to anything besides reporting final value, you cannot base any decisions on it.

Answer 2

It depends on what you are doing.

If you are using the test accuracy to fit hyper-parameters, then yes, you are cheating. The test data is "leaking" into your training data indirectly and you are overfitting.

Thus, it is advisable to have 3 datasets: Train, Test and Validation data.

Lock the Test data away, do not touch it till you take hands off tweaking your model (training).

Use the validation data as much as you like to find the best hyperparameters etc.

When you are done, unlock the Test data to report the accuracy of your model (and just to emphasize, do not use this feedback to tweak your model).

If you are not happy with the accuracy you get on the test data, consider rethinking your model. If you are not happy with the test data results, you won't be happy once the model is in production.

If you don't have enough data to create three partitions, consider doing k-fold cross validation.