ELU (Exponential Linear Unit) takes a different approach to fixing ReLU's negative-side flatness: instead of a straight negative slope like Leaky ReLU, it uses a smooth exponential curve that saturates gently toward a fixed negative value.
Formula
\(\alpha\) is typically 1.0. As \(z\to-\infty\), \(\text{ELU}(z)\to-\alpha\) — the curve smoothly flattens toward a fixed negative floor rather than continuing linearly downward like Leaky ReLU.
Derivative
Graph
A smooth exponential curve for negative inputs, flattening toward -α instead of continuing linearly downward.
Numerical Example
With \(\alpha=1\): \(\text{ELU}(-1) = 1(e^{-1}-1) \approx 1(0.368-1) = -0.632\). \(\text{ELU}(-5) = 1(e^{-5}-1) \approx -0.993\) — approaching, but never quite reaching, \(-1\).
Why the Smooth Curve Matters — Mean Activation Closer to Zero
ReLU's outputs are always \(\ge 0\), so the average activation across a layer tends to be positive — a mild version of the "not zero-centered" issue that sigmoid also has (see Sigmoid Function). ELU's negative outputs push the mean activation closer to zero, which empirically tends to speed up learning by keeping the gradient's overall direction less biased. ELU is also smooth (differentiable) everywhere, including at \(z=0\), unlike ReLU's sharp corner there.
Code
import numpy as np
import torch.nn as nn
import torch
def elu(z, alpha=1.0):
return np.where(z >= 0, z, alpha * (np.exp(z) - 1))
print(elu(np.array([-5, -1, 0, 5]))) # [-0.993 -0.632 0. 5. ]
layer = nn.ELU(alpha=1.0)
print(layer(torch.tensor([-5.0, -1.0, 0.0, 5.0])))The Tradeoff: Computational Cost
ELU requires computing \(e^z\) for every negative activation — meaningfully more expensive than ReLU's simple comparison-and-max, similar to sigmoid/tanh's cost. This is the main reason ReLU (or Leaky ReLU) remains the more common default despite ELU's sometimes-better convergence properties — the computational tradeoff isn't always worth it at scale.
Common Mistakes
- Assuming ELU always trains faster than ReLU in wall-clock time — it may converge in fewer epochs, but each epoch costs more compute due to the exponential, and the net effect depends heavily on the specific task and hardware.
Interview Relevance
Q: "What advantage does ELU have over ReLU, and what does it cost?" ELU's negative-side outputs push the mean activation closer to zero (unlike ReLU, whose outputs are always non-negative), which can speed up convergence, and it's smooth everywhere unlike ReLU's kink at zero. The cost is computing an exponential for every negative input, which is more expensive than ReLU's simple max operation.
Practice Question
As \(z \to -\infty\), what value does ELU(z) approach, given \(\alpha=1\)? Why does this "floor" behavior differ from Leaky ReLU's unbounded negative output?
