Huber Loss is a deliberate compromise between MSE and MAE — quadratic (like MSE) for small errors, so gradients stay smooth near the optimum, and linear (like MAE) for large errors, so outliers don't dominate training.
Formula
\(\delta\) is a threshold hyperparameter marking where the loss transitions from quadratic to linear behavior.
Visualizing the Two Regimes
Huber loss matches MSE's smooth curve near zero error, then switches to MAE's linear growth beyond δ — bounding the influence of large errors.
Numerical Example
With \(\delta=1\): for error \(0.5\) (\(\le\delta\)): \(L = \frac{1}{2}(0.5)^2 = 0.125\). For error \(3\) (\(>\delta\)): \(L = 1(3-0.5) = 2.5\). Compare to what pure squared error would give for the second case: \(\frac{1}{2}(3)^2=4.5\) — Huber's linear regime produces a notably smaller penalty for this large error, exactly the outlier-dampening effect it's designed for.
Choosing \(\delta\)
| \(\delta\) Value | Behavior |
|---|---|
| Very small | Behaves almost entirely like MAE — robust to outliers, less smooth gradient near zero |
| Very large | Behaves almost entirely like MSE — smooth everywhere, sensitive to outliers |
| Moderate (task-dependent) | Genuine compromise — the typical, intended use case |
Code
import torch.nn as nn
import torch
loss_fn = nn.HuberLoss(delta=1.0)
y_true = torch.tensor([10.0, 20.0, 30.0])
y_pred = torch.tensor([10.5, 20.0, 33.0]) # small error, zero error, large error
print(loss_fn(y_pred, y_true))
# PyTorch's SmoothL1Loss is closely related (a Huber variant with a fixed
# implicit delta=1, historically used in object detection bounding-box regression)
smooth_l1 = nn.SmoothL1Loss()
print(smooth_l1(y_pred, y_true))Where It's Used Today
Common in object detection bounding-box regression (as Smooth L1 Loss) and any regression task where a mix of typical, well-behaved errors and occasional large outlier errors is expected — a common situation in real-world sensor or labeling data.
Common Mistakes
- Treating \(\delta\) as a value that doesn't need tuning — the right threshold genuinely depends on your target's scale and expected error distribution; a poorly chosen \(\delta\) can behave almost identically to plain MSE or plain MAE, losing the intended compromise.
Interview Relevance
Q: "What problem does Huber Loss solve that neither pure MSE nor pure MAE solves well on its own?" It combines MSE's smooth, well-behaved gradient near the optimum (helping stable convergence) with MAE's bounded, linear penalty for large errors (limiting the influence of outliers) — giving you both properties by switching behavior at a threshold \(\delta\), rather than forcing a single tradeoff across the whole error range.
Practice Question
With \(\delta=2\), compute the Huber loss for an error of \(1\) and for an error of \(5\). Which regime (quadratic or linear) does each fall into?
