Leaky ReLU is ReLU with one small but important change: instead of outputting exactly 0 for negative inputs, it outputs a small negative slope. That tiny change is enough to prevent neurons from dying permanently.
Formula
\(\alpha\) is a small constant, commonly 0.01, fixed before training (a hyperparameter, not learned).
Derivative
Graph
A shallow negative slope (exaggerated here for visibility) instead of a flat zero for negative inputs.
Why This Fixes Dying ReLU
Because the derivative for negative inputs is \(\alpha\) (a small but non-zero number) instead of exactly 0, a neuron with a persistently negative weighted sum still receives a (small) gradient signal and can still update its weights, potentially recovering into a useful state. This directly resolves the dying ReLU problem from ReLU, at the cost of one extra hyperparameter (\(\alpha\)) to set.
Numerical Example
Code
import numpy as np
import torch.nn as nn
import torch
def leaky_relu(z, alpha=0.01):
return np.where(z >= 0, z, alpha * z)
print(leaky_relu(np.array([-5, 0, 5]))) # [-0.05 0. 5. ]
layer = nn.LeakyReLU(negative_slope=0.01)
print(layer(torch.tensor([-5.0, 0.0, 5.0])))Common Mistakes
- Assuming Leaky ReLU always outperforms plain ReLU — in practice the difference is often small, and ReLU's simplicity/speed still makes it a reasonable default; Leaky ReLU is worth trying specifically if dying neurons are observed to be a real problem.
- Setting \(\alpha\) too large — if \(\alpha\) approaches 1, the function approaches a plain linear activation, losing the non-linearity a network needs (see Linear Transformations).
Interview Relevance
Q: "How does Leaky ReLU solve the dying ReLU problem?" By giving negative inputs a small non-zero slope (\(\alpha\), typically 0.01) instead of flattening them to exactly 0. This means a neuron with a negative weighted sum still has a non-zero gradient and can continue to update its weights during training, instead of getting permanently stuck outputting zero.
Practice Question
With \(\alpha=0.1\), compute Leaky ReLU's output and derivative at \(z=-4\).
