PyTorch Validation Loop

The practical, complete PyTorch validation loop — building on Validation Loop's conceptual coverage, with the exact syntax and a full accuracy-tracking example.

The Complete Loop

import torch

def validate(model, val_loader, loss_fn, device):
    model.eval()               # disables dropout, uses running batch-norm statistics
    total_loss = 0.0
    correct = 0
    total = 0

    with torch.no_grad():        # no gradient tracking needed -- saves memory and compute
        for x_batch, y_batch in val_loader:
            x_batch, y_batch = x_batch.to(device), y_batch.to(device)

            predictions = model(x_batch)
            loss = loss_fn(predictions, y_batch)
            total_loss += loss.item() * x_batch.size(0)   # weight by batch size for a correct average

            predicted_classes = predictions.argmax(dim=1)
            correct += (predicted_classes == y_batch).sum().item()
            total += y_batch.size(0)

    avg_loss = total_loss / total
    accuracy = correct / total
    return avg_loss, accuracy

Why Weight by Batch Size When Averaging Loss

If the last batch is smaller than the others (a common outcome when the dataset size isn't evenly divisible by batch_size), simply averaging each batch's already-averaged loss treats every batch equally regardless of size — subtly skewing the overall average. Multiplying each batch's loss by its actual size before summing, then dividing by the true total example count, gives the mathematically correct overall average.

Integrating Into the Full Training Loop

for epoch in range(num_epochs):
    model.train()
    for x_batch, y_batch in train_loader:
        # ... standard training step ...
        pass

    val_loss, val_accuracy = validate(model, val_loader, loss_fn, device)
    print(f"Epoch {epoch+1}: val_loss={val_loss:.4f}, val_accuracy={val_accuracy:.4f}")

Common Mistakes

• Forgetting model.eval() before validation — dropout stays active and batch norm uses batch statistics instead of running statistics, producing noisy, inconsistent validation metrics.
• Computing accuracy using .argmax(dim=1) on already-softmaxed probabilities versus raw logits — this actually gives the identical answer (softmax preserves relative ordering), but mixing up which one you're working with elsewhere in the code can cause confusion about what values represent.

Interview Relevance

Q: "Why does a correct validation loss average need to weight each batch's loss by its batch size, rather than just averaging batch-level averages?" If batch sizes vary (typically only the final batch, when the dataset size isn't evenly divisible), naively averaging the batch-level averages treats every batch equally regardless of how many examples it actually contains — subtly skewing the result. Weighting each batch's loss contribution by its actual size before dividing by the total example count produces the mathematically correct, example-level average.

Practice Question

Why is torch.no_grad() used during validation, and what would happen (besides wasted memory) if it were omitted?

Related DL Notes

• Validation Loop
• PyTorch Training Loop
• Early Stopping