Pytorch: Intermediate testing during training

Question

How can I test my pytorch model on validation data during training? I know that there is the function myNet.eval() which apparantly switches of any dropout layers, but is it also preventing the gradients from being accumulated?
Also how would I undo the myNet.eval() command in order to continue with the training?

If anyone has some code snippet / toy example I would be grateful!

Answer 1

How can I test my pytorch model on validation data during training?

There are plenty examples where there are train and test steps for every epoch during training. An easy one would be the official MNIST example. Since pytorch does not offer any high-level training, validation or scoring framework you have to write it yourself. Commonly this consists of

• a data loader (commonly based on torch.utils.dataloader.Dataloader)
• a main loop over the total number of epochs
• a train() function that uses training data to optimize the model
• a test() or valid() function to measure the effectiveness of the model given validation data and a metric

This is also what you will find in the linked example.

Alternatively you can use a framework that provides basic looping and validation facilities so you don't have to implement everything by yourself all the time.

• tnt is torchnet for pytorch, supplying you with different metrics (such as accuracy) and abstraction of the train loop. See this MNIST example.
• inferno and torchsample attempt to model things very similar to Keras and provide some tools for validation
• skorch is a scikit-learn wrapper for pytorch that lets you use all the tools and metrics from sklearn

Also how would I undo the myNet.eval() command in order to continue with the training?

myNet.train() or, alternatively, supply a boolean to switch between eval and training: myNet.train(True) for train mode.