Is it ok to only use one epoch?

Question

I'm training a neural network in TensorFlow (using tflearn) on data that I generate. From what I can tell, each epoch we use all of the training data. Since I can control how many examples I have, it seems like it would be best to just generate more training data until one epoch is enough to train the network.

So my question is: Is there any downside to only using one epoch, assuming I have enough training data? Am I correct in assuming that 1 epoch of a million examples is better than 10 epochs of 100,000?

Answer 1

Following a discussion with @Prune:

Suppose you have the possibility to generate an infinite number of labeled examples, sampled from a fixed underlying probability distribution, i.e. from the same manifold.

The more examples the network see, the better it will learn, and especially the better it will generalize. Ideally, if you train it long enough, it could reach 100% accuracy on this specific task.

---

The conclusion is that only running 1 epoch is fine, as long as the examples are sampled from the same distribution.

The limitations to this strategy could be:

• if you need to store the generated examples, you might run out of memory
• to handle unbalanced classes (cf. @jorgemf answer), you just need to sample the same number of examples for each class.
  • e.g. if you have two classes, with 10% chance of sampling the first one, you should create batches of examples with a 50% / 50% distribution

---

it's possible that running multiple epochs might make it learn some uncommon cases better.

I disagree, using multiple times the same example is always worse than generating new unknown examples. However, you might want to generate harder and harder examples with time to make your network better on uncommon cases.

Answer 2

You need training examples in order to make the network learn. Usually you don't have so many examples in order to make the network converge, so you need to run more than one epoch.

It is ok to use only one epoch if you have so many examples and they are similar. If you have 100 classes but some of them only have very few examples you are not going to learn those classes only with one epoch. So you need balanced classes.

Moreover, it is a good idea to have a variable learning rate which decreases with the number of examples, so the network can fine tune itself. It starts with a high learning rate and then decreases it over time, if you only run for one epoch you need to bear in mind this to tweak the graph.

My suggestion is to run more than one epoch, mostly because the more examples you have the more memory you need to store them. But if memory is fine and learning rate is adjusted based on number of examples and not epochs, then it is fine run one epoch.

Edit: I am assuming you are using a learning algorithm which updates the weights of the network every batch or similar.