Practical guidance for choosing which activation function to use where — synthesizing the comparison notes from the Activation Functions category into direct, actionable recommendations.
The Practical Defaults by Layer Type and Architecture
| Where | Default Choice | Why |
|---|---|---|
| Hidden layers, CNNs/MLPs | ReLU | Cheap, non-saturating for positive inputs, well-understood, strong empirical track record — see ReLU |
| Hidden layers, Transformers | GELU | Empirically favored in this specific regime, standard in BERT/GPT-family feed-forward blocks — see ReLU vs GELU |
| Binary classification output | Sigmoid | Directly produces \(P(\text{class}=1)\) — see Sigmoid Function |
| Multi-class classification output | Softmax | Produces a valid probability distribution over mutually exclusive classes — see Softmax Function |
| Regression output | None (linear/identity) | Allows unbounded output values — see Linear Activation |
| LSTM/GRU gates | Sigmoid (gates), Tanh (candidate states) | Bounded ranges carry specific semantic meaning here — see Sigmoid vs Tanh |
When to Deviate From ReLU in Hidden Layers
If training diagnostics reveal a significant fraction of "dead" neurons (see the dying ReLU problem from ReLU), trying Leaky ReLU or ELU (see ReLU vs Leaky ReLU) is a reasonable, targeted adjustment rather than a default choice made preemptively without evidence of the specific problem.
Code — Diagnosing Dead ReLU Neurons
import torch
def check_dead_relus(model, val_loader):
activation_counts = {}
hooks = []
def make_hook(name):
def hook(module, input, output):
activation_counts[name] = activation_counts.get(name, 0) + (output > 0).float().mean().item()
return hook
for name, module in model.named_modules():
if isinstance(module, torch.nn.ReLU):
hooks.append(module.register_forward_hook(make_hook(name)))
for x, _ in val_loader:
model(x)
break # a single batch is often enough for a quick diagnostic
for h in hooks:
h.remove()
return activation_counts # low values suggest a large fraction of "dead" (always-zero) activationsCommon Mistakes
- Reflexively using a variant like Leaky ReLU or GELU everywhere "just in case," without evidence the plain default is actually causing a problem — this adds unnecessary complexity and (for GELU specifically) compute cost without a clear justification.
- Using sigmoid or tanh for hidden layers in a deep feedforward network by default — as covered extensively in the Activation Functions category, this reintroduces vanishing gradients that ReLU-family activations largely resolved.
Interview Relevance
Q: "Why would you choose GELU for a Transformer-based model's hidden layers but ReLU for a CNN's hidden layers?" This reflects an empirical, architecture-specific finding rather than a universal ranking — GELU's smooth, non-monotonic shape has been found to help optimization stability specifically in the deep, large-scale, self-attention-heavy regime of Transformers (established starting with BERT), while CNNs, more sensitive to per-operation compute cost applied at every spatial location, generally haven't shown a strong enough benefit from GELU to justify replacing ReLU's simplicity and speed.
Practice Question
You're building the output layer for a model predicting a house's price (a continuous, unbounded positive value). What activation function (if any) would you use, and why?
