PyTorch: Shuffle DataLoader

Question

There are several scenarios that make me confused about shuffling the data loader, which are as follows.

I set the “shuffle” parameter to False on both train_loader and valid_loader. then the results I get are as follows

Epoch 1/4    loss=0.8802     val_loss=0.8202     train_acc=0.63      val_acc=0.63  
Epoch 2/4    loss=0.6993     val_loss=0.6500     train_acc=0.66      val_acc=0.72 
Epoch 3/4    loss=0.5363     val_loss=0.5385     train_acc=0.76      val_acc=0.80
Epoch 4/4    loss=0.4055     val_loss=0.5130     train_acc=0.85      val_acc=0.81

I set the “shuffle” parameter to True on train_loader and False to valid_loader. then the results I get are as follows

Epoch 1/4    loss=0.8928     val_loss=0.8284     train_acc=0.63      val_acc=0.63 
Epoch 2/4    loss=0.7308     val_loss=0.6263     train_acc=0.61      val_acc=0.73 
Epoch 3/4    loss=0.5594     val_loss=0.5046     train_acc=0.54      val_acc=0.81 
Epoch 4/4    loss=0.4304     val_loss=0.4525     train_acc=0.49      val_acc=0.82 

Based on that result, my training accuracy has a worse performance when I shuffle train_loader.

And this is a snippet of my code.

for epoch in range(n_epochs):
    model.train()
    avg_loss = 0.
    train_preds = np.zeros((len(train_X),len(le.classes_)))

    for i, (x_batch, y_batch) in enumerate(train_loader):
        y_pred = model(x_batch)
        loss = loss_fn(y_pred, y_batch)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        avg_loss += loss.item() / len(train_loader)
        train_preds[i * batch_size:(i+1) * batch_size] = F.softmax(y_pred).cpu().detach().numpy()            

    train_accuracy = sum(train_preds.argmax(axis=1) == y_train)/len(y_train)

    model.eval()        
    avg_val_loss = 0.
    val_preds = np.zeros((len(x_cv),len(le.classes_)))

    for i, (x_batch, y_batch) in enumerate(valid_loader):
        y_pred = model(x_batch).detach()
        avg_val_loss += loss_fn(y_pred, y_batch).item() / len(valid_loader)
        val_preds[i * batch_size:(i+1) * batch_size] =F.softmax(y_pred).cpu().numpy()
    val_accuracy = sum(val_preds.argmax(axis=1)==y_test)/len(y_test)

Did I make a mistake calculating the training accuracy? Thanks in advance

Answer 1

You are comparing the shuffled predictions with the un-shuffled labels. To fix that, count the number of accurate predictions in every iteration, and compute the overall accuracy at the end.

for epoch in range(n_epochs):
    model.train()
    avg_loss = 0.
    total_correct = 0
    total_samples = 0

    for i, (x_batch, y_batch) in enumerate(train_loader):
        y_pred = model(x_batch)
        loss = loss_fn(y_pred, y_batch)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        avg_loss += loss.item() / len(train_loader)
        total_correct += (torch.argmax(y_pred, 1) == y_batch).sum()
        total_samples += y_batch.shape[0]

    train_accuracy = total_correct / total_samples

(I haven't tested this code)