Batch Gradient Descent computes the gradient using the entire training dataset before making a single weight update. It's the most accurate variant of gradient descent — and the least practical one for real-world dataset sizes.
Formula
Every one of the \(N\) training examples contributes to the gradient before \(\mathbf{w}\) is updated even once.
Advantages and Disadvantages
| Advantage | Disadvantage |
|---|---|
| Gradient is exact and stable — no noise from sampling | Extremely slow: one full pass over the entire dataset per single weight update |
| Convergence path is smooth, easy to reason about | Impractical memory requirements for large datasets (must hold gradients for all N examples) |
| Deterministic — same result every run given the same starting point | Can get stuck in a poor local minimum, with no noise to help escape it |
Why It's Rarely Used in Practice
For a dataset with millions of examples, computing just one gradient update requires processing the entire dataset — with modern datasets, this means a single weight update could take minutes to hours, and you'd need many thousands of updates to actually train a model. This impracticality is exactly what motivates Stochastic and Mini-Batch Gradient Descent, covered next.
Code
import torch
# Batch gradient descent: ALL data used for every single update
def batch_gradient_descent(X, y, w_init, lr=0.01, epochs=100):
w = w_init.clone().requires_grad_(True)
for epoch in range(epochs):
predictions = X @ w
loss = ((predictions - y) ** 2).mean() # uses the FULL dataset X, y every time
loss.backward()
with torch.no_grad():
w -= lr * w.grad
w.grad.zero_()
return w
X = torch.randn(10000, 5) # 10,000 examples -- every epoch processes all of them before updating
y = torch.randn(10000)
w = batch_gradient_descent(X, y, torch.zeros(5, requires_grad=True))Common Mistakes
- Assuming "batch" in "batch gradient descent" means the same thing as a "batch" in mini-batch training — it doesn't; here "batch" specifically means the entire dataset, the opposite of the small batches used in mini-batch gradient descent and PyTorch's DataLoader.
- Using batch gradient descent by default for large modern datasets — this is essentially never done in practice for deep learning; mini-batch is the near-universal standard.
Interview Relevance
Q: "Why isn't batch gradient descent used for training modern deep learning models?" It requires computing gradients over the entire dataset before a single weight update, which is computationally impractical for datasets with millions of examples — both in time (one update could take extremely long) and memory. Mini-batch gradient descent achieves nearly the same gradient quality with vastly more frequent, faster updates.
Practice Question
If a dataset has 1 million examples and batch gradient descent is used, how many weight updates happen per epoch? How does this compare to mini-batch gradient descent with a batch size of 256?
