ReLU (Rectified Linear Unit) is the default activation function for hidden layers in modern deep learning. Its formula is almost embarrassingly simple — and that simplicity is exactly why it solves the vanishing gradient problem that limited sigmoid and tanh.
Formula
Derivative
| Property | Value |
|---|---|
| Range | \([0, \infty)\) |
| Derivative for \(z>0\) | Exactly 1 — no saturation, no shrinkage |
Graph
Zero for any negative input, then a straight line with slope 1 for positive inputs — no saturation on the positive side.
Why ReLU Fixes Vanishing Gradients
For any positive input, ReLU's derivative is exactly 1 — not a fraction less than 1 like sigmoid or tanh's derivative almost everywhere. When backpropagation multiplies gradients across many ReLU layers (for the "active" neurons), the product doesn't shrink from this activation's own derivative — it's multiplied by 1 repeatedly. This directly addresses the mechanism behind vanishing gradients described in Sigmoid Function, and it's the single biggest reason ReLU enabled training much deeper networks than were previously practical (see Why Deep Learning Became Successful).
Computational Efficiency — A Second Major Advantage
ReLU is just a comparison and a max operation — dramatically cheaper to compute than sigmoid or tanh, which both require an exponential. At the scale of billions of activations computed per training step in a large network, this efficiency difference is significant in real training time.
The Dying ReLU Problem
ReLU's own flaw: if a neuron's weights update such that its weighted sum \(z\) is consistently negative for every input in the training data, its output is always 0, and its gradient is always 0 — it stops learning entirely, permanently. This neuron is "dead." A large enough learning rate, or a large negative gradient update, can push a neuron into this state, and it never recovers because zero gradient means zero further updates. This exact problem motivates Leaky ReLU, PReLU and ELU, covered in the following notes.
Numerical Example
Code
import numpy as np
import torch
import torch.nn as nn
def relu(z):
return np.maximum(0, z)
print(relu(np.array([-3, 0, 5]))) # [0 0 5]
z = torch.tensor([-3.0, 0.0, 5.0])
print(torch.relu(z)) # functional form
layer_activation = nn.ReLU() # module form, common inside nn.Sequential
print(layer_activation(z))Where It's Used Today
The default choice for hidden layers across CNNs, MLPs, and many other architectures. It's so standard that "use ReLU unless you have a specific reason not to" is reasonable practical advice for hidden layers in most feedforward and convolutional networks.
Common Mistakes
- Using ReLU in the output layer for tasks where negative outputs are valid (e.g. general regression) — ReLU's range is \([0,\infty)\), so it mathematically cannot produce a negative prediction.
- Setting an aggressively high learning rate without noticing a growing number of "dead" (always-zero) neurons over training — monitoring activation statistics can catch this early.
Interview Relevance
Q: "What is the 'dying ReLU' problem, and what causes it?" A ReLU neuron whose weighted sum ends up negative for every training input outputs 0 and has zero gradient — it stops learning permanently, since no gradient update can change its dead state. This can happen from a large negative weight update (e.g. from too high a learning rate), and is one motivation for variants like Leaky ReLU that allow a small non-zero gradient for negative inputs.
Practice Question
Explain, using ReLU's derivative, why a "dead" ReLU neuron can never recover during training via gradient descent alone.
