Validation Loop

The validation loop runs alongside training — typically once per epoch — to measure how well the model generalizes to data it hasn't been trained on, using no gradient computation at all.

The Complete Structure

model.eval()                     # sets the model to EVALUATION mode
val_loss = 0.0
correct = 0

with torch.no_grad():             # disables gradient tracking -- saves memory and compute
    for X_batch, y_batch in val_loader:
        predictions = model(X_batch)
        loss = loss_fn(predictions, y_batch)
        val_loss += loss.item()
        correct += (predictions.argmax(dim=1) == y_batch).sum().item()

avg_val_loss = val_loss / len(val_loader)
val_accuracy = correct / len(val_loader.dataset)
print(f"Validation: loss={avg_val_loss:.4f}, accuracy={val_accuracy:.4f}")

Two Critical Differences From the Training Loop

Crucially, there's no loss.backward(), no optimizer.step(), and no optimizer.zero_grad() anywhere in a validation loop — the model's weights are never touched during validation; it's purely a read-only measurement.

Why Validation Runs Alongside Training, Not Just at the End

Tracking validation loss/accuracy every epoch (alongside training loss/accuracy) is exactly what reveals overfitting as it happens: if training loss keeps decreasing but validation loss starts increasing, the model is beginning to memorize training-specific noise instead of learning generalizable patterns — precisely the signal Early Stopping and Checkpointing are built to respond to.

Visualizing the Signal

Common Mistakes

• Forgetting model.eval() before validation — dropout stays active and batch norm keeps using batch-level statistics, producing noisy, misleadingly inconsistent validation metrics from run to run.
• Forgetting torch.no_grad() — validation will still work correctly, but wastes memory and compute unnecessarily caching values that will never be used for a backward pass.
• Accidentally calling optimizer.step() or loss.backward() inside a validation loop — this would leak validation data into training, corrupting the entire point of having a separate validation set.

Interview Relevance

Q: "What would happen if you forgot to call model.eval() before running validation?" Dropout would remain active, randomly zeroing activations during validation just as it does during training — and batch normalization would use the current (validation) batch's statistics instead of its learned running averages. Both effects introduce noise and inconsistency into validation metrics, making them unreliable for judging true generalization performance or comparing across epochs.

Practice Question

Why is it safe and correct to disable gradient tracking (torch.no_grad()) during validation, but not during training?

Related DL Notes

• Training Loop
• Overfitting
• Early Stopping