L1 regularization (Lasso) penalizes the sum of coefficients' absolute values — its distinctive, genuinely useful property is that it can shrink some coefficients to exactly zero, performing automatic feature selection as a side effect of training.
Formula
Worked Example — Soft Thresholding
For a simplified single-coefficient case, L1-regularized regression has a closed-form "soft thresholding" solution:
Starting from an unregularized coefficient \(w_{\text{OLS}}=5\):
| λ | Calculation | Result |
|---|---|---|
| 6 | \(\max(5 - 3, 0) = 2\) | 2 (shrunk, still nonzero) |
| 12 | \(\max(5 - 6, 0) = \max(-1, 0)\) | 0 (exactly zero!) |
Once the penalty (\(\lambda/2 = 6\)) exceeds the coefficient's original magnitude (5), the coefficient is pushed all the way to exactly zero — this "corner" behavior is unique to L1, and is exactly why it performs feature selection.
from sklearn.linear_model import Lasso
import numpy as np
X = np.random.rand(100, 20) # 20 features, only 3 actually matter
y = 3*X[:,0] - 2*X[:,1] + 5*X[:,2] + np.random.normal(0, 0.1, 100)
model = Lasso(alpha=0.1)
model.fit(X, y)
print(np.round(model.coef_, 3))
print("Nonzero coefficients:", np.sum(model.coef_ != 0)) # often close to 3 -- Lasso found themWhy the Diamond Shape Matters — The Geometric Explanation
L1's constraint region (\(\sum|w_i| \leq t\)) is a diamond in coefficient space, with sharp corners aligned exactly on the axes. Because the cost function's optimum tends to land on a corner of this constraint region when the two shapes intersect, and corners correspond to some coefficients being exactly zero, L1 solutions naturally end up sparse. L2's circular constraint region has no corners, so its solutions shrink toward — but essentially never exactly reach — zero.
Practical Use Cases
- High-dimensional data where you suspect only a subset of features are genuinely relevant
- Wanting both a predictive model AND automatic feature selection from a single training run — see Embedded Methods
Common Mistakes
- Using L1 when features are highly correlated with each other — Lasso tends to arbitrarily pick one from a correlated group and zero out the rest, which can be unstable (small data changes shift which one "wins").
- Not scaling features first, same requirement as any regularization technique.
Interview Relevance
Q: "Why does L1 produce exactly-zero coefficients while L2 doesn't?" L1's penalty (sum of absolute values) creates a constraint region with sharp corners on the coordinate axes — the optimal solution often lands exactly on one of these corners, where some coefficients are precisely zero; L2's penalty (sum of squares) creates a smooth circular region with no corners, so its optimal solutions shrink toward zero without typically reaching it exactly.
Practice Question
Using the soft-thresholding formula, compute the Lasso-regularized coefficient for \(w_{\text{OLS}}=8\) with \(\lambda=10\).
