L1 regularization adds a penalty proportional to the sum of the absolute values of the weights — and its distinctive effect, thanks to the L1 norm's geometry (see Vector Norms), is pushing some weights all the way to exactly zero.
Formula
This is exactly \(\lambda\|\mathbf{w}\|_1\), the L1 norm scaled by \(\lambda\), added directly to the loss.
Why L1 Produces Sparsity
The gradient of \(|w_i|\) with respect to \(w_i\) is \(\pm1\) (a constant, regardless of how large or small \(w_i\) already is) — every gradient step shrinks a weight by the same fixed amount, pushing small weights all the way to exactly zero rather than just shrinking them proportionally. This is the same L1-vs-L2 geometry difference illustrated in Vector Norms's diamond-vs-circle diagram — L1's sharp corners on the coordinate axes are exactly where the optimal solution tends to land, with some weight coordinates at exactly zero.
Numerical Example
A weight \(w=0.05\) under L1 regularization with gradient contribution \(\lambda\cdot\text{sign}(w) = \lambda\): with \(\lambda=0.1\) and learning rate \(\eta=0.5\), the L1 penalty alone pulls the weight down by \(0.5\times0.1=0.05\) every step — enough to push this particular small weight to exactly 0 in a single step, and keep it there (since \(\text{sign}(0)\) is typically treated as 0, no further penalty pushes it past zero).
Code
import torch
def l1_penalty(model, lam=0.01):
return lam * sum(p.abs().sum() for p in model.parameters())
# Added directly to the loss before calling .backward()
loss = loss_fn(y_pred, y_true) + l1_penalty(model)
loss.backward()Where Sparsity Is Valuable
Sparse weights (many exactly zero) can act as an implicit form of feature selection — if an input feature's associated weight is driven to exactly zero, that feature is effectively excluded from the model's decision. This makes L1 particularly useful when many input features are suspected to be irrelevant or redundant, and a sparser, more interpretable model is desirable.
Common Mistakes
- Expecting L1 regularization to behave identically to L2 with just a different formula — the sparsity-inducing behavior is a genuinely distinct qualitative effect, not just a matter of degree.
- Using an L1 penalty strength so large that it drives too many weights to zero, effectively disabling large portions of the network — as with any regularization strength, \(\lambda\) needs tuning, not an arbitrary large default.
Interview Relevance
Q: "Why does L1 regularization tend to produce sparse weights while L2 doesn't?" L1's penalty gradient is a constant (\(\pm1\)) regardless of a weight's current magnitude, so it shrinks small weights by a fixed amount each step until they hit exactly zero. L2's penalty gradient is proportional to the weight's own value, so it shrinks weights geometrically — approaching zero but rarely landing exactly on it. This difference traces directly back to the geometric shape of the L1 vs L2 norm's constraint region, covered in Vector Norms.
Practice Question
A dataset has 200 input features, but domain knowledge suggests only about 20 are actually relevant. Would L1 or L2 regularization be more naturally suited to this situation, and why?
