Focal Loss modifies cross-entropy with one added factor that automatically down-weights easy, already-well-classified examples — letting training focus its gradient signal on the hard, still-misclassified ones. It was introduced specifically to fix extreme class imbalance in object detection.
Formula
\(p_t\) is the model's predicted probability for the true class (matching standard cross-entropy's \(\hat y_y\)). \(\gamma \ge 0\) (commonly 2) is the focusing parameter; \(\alpha\) is an optional class-balancing weight. When \(\gamma=0\), Focal Loss reduces exactly to standard cross-entropy.
The Key Mechanism: \((1-p_t)^\gamma\)
| Prediction Confidence | \(p_t\) | \((1-p_t)^\gamma\) with \(\gamma=2\) | Effect |
|---|---|---|---|
| Easy, confidently correct | 0.9 | \((0.1)^2=0.01\) | Loss scaled down to 1% of standard cross-entropy — barely contributes to training |
| Hard, uncertain | 0.5 | \((0.5)^2=0.25\) | Loss scaled down modestly — still contributes meaningfully |
| Very hard, confidently wrong | 0.1 | \((0.9)^2=0.81\) | Loss barely reduced — dominates the gradient signal |
Numerical Example
With \(\gamma=2\), \(\alpha=1\): for \(p_t=0.9\) (easy example): \(\text{FL} = -(0.1)^2\log(0.9) \approx -0.01(-0.105) = 0.00105\), versus plain cross-entropy's \(-\log(0.9)\approx0.105\) — a 100x reduction. For \(p_t=0.1\) (hard example): \(\text{FL} = -(0.9)^2\log(0.1) \approx -0.81(-2.303)=1.865\), versus plain cross-entropy's \(2.303\) — only a modest reduction. The hard example's relative contribution to the total loss grows enormously compared to the easy one.
Why This Matters for Class Imbalance
In object detection, background regions (no object present) vastly outnumber actual object regions — often by a ratio of thousands to one. Standard cross-entropy, summed over every region, ends up dominated by the sheer volume of easy background examples, drowning out the gradient signal from the rare, informative object examples. Focal Loss's down-weighting of easy examples directly counteracts this — it was introduced specifically alongside the RetinaNet object detector to solve exactly this problem.
Code
import torch
import torch.nn.functional as F
def focal_loss(logits, targets, alpha=1.0, gamma=2.0):
ce_loss = F.cross_entropy(logits, targets, reduction='none')
p_t = torch.exp(-ce_loss) # recovers p_t from the cross-entropy value
focal = alpha * (1 - p_t) ** gamma * ce_loss
return focal.mean()
logits = torch.tensor([[2.0, 0.1], [0.1, 2.0]])
targets = torch.tensor([0, 1])
print(focal_loss(logits, targets))Common Mistakes
- Applying Focal Loss to a balanced dataset by default — its benefit is specifically for severe class imbalance; on balanced data it mostly just changes the effective learning rate for easy vs. hard examples without addressing an actual imbalance problem.
- Choosing \(\gamma\) without any tuning — \(\gamma=2\) is a common starting point from the original paper, but the right value depends on how extreme your specific class imbalance is.
Interview Relevance
Q: "How does Focal Loss address class imbalance differently from simply reweighting classes?" Simple class reweighting (like \(\alpha\) alone) adjusts the loss based on which class an example belongs to — a fixed weight per class. Focal Loss additionally reweights based on how hard each individual example currently is (via \((1-p_t)^\gamma\)), dynamically down-weighting easy examples regardless of class, which directly targets the "many easy examples drown out few hard ones" problem that class-level reweighting alone doesn't fully solve.
Practice Question
With \(\gamma=2\), compare the scaling factor \((1-p_t)^\gamma\) for \(p_t=0.99\) versus \(p_t=0.3\). What does this tell you about how much Focal Loss "cares" about each example during training?
