The linear activation function simply returns its input unchanged. It's the simplest possible activation — and understanding exactly why it's almost never used between hidden layers reinforces the core lesson from Linear Transformations.
Formula
| Property | Value |
|---|---|
| Range | \((-\infty, \infty)\) |
| Derivative | \(\phi'(z) = 1\) — constant, everywhere |
Graph
Output equals input exactly — a straight line through the origin with slope 1.
Why It Can't Be Used Between Hidden Layers
This is the exact scenario proven in Linear Transformations: stacking layers that each use a linear activation collapses the entire network into a single linear transformation, regardless of depth. A 50-layer network with linear activations throughout has no more representational power than a single linear layer — depth becomes completely wasted.
Where It Actually Is Used: Regression Output Layers
The one place a linear (identity) activation is standard is the output layer of a regression network — where you want the raw, unbounded weighted sum as the final prediction (e.g. predicting a house price, which can be any positive real number, or even a temperature, which can be negative). Squashing this final output through a bounded activation like sigmoid or tanh would artificially cap what the network could ever predict.
Code
import torch.nn as nn
# Regression network: hidden layers use ReLU, but the OUTPUT layer uses no
# activation (equivalent to a linear/identity activation) -- this is standard practice
model = nn.Sequential(
nn.Linear(10, 64), nn.ReLU(),
nn.Linear(64, 32), nn.ReLU(),
nn.Linear(32, 1) # no activation here = linear output, suited to unbounded regression targets
)Common Mistakes
- Using linear activation on hidden layers "to keep things simple" — this silently collapses the network's effective depth to 1 layer, wasting every parameter in the discarded layers.
- Forgetting that not specifying an activation after a layer in most frameworks (like the final
nn.Linearabove) is exactly a linear/identity activation — it's an intentional choice for regression outputs, not an oversight.
Interview Relevance
Q: "Why does a regression model's final layer typically have no activation function, or a linear one?" Because the target values (e.g. price, temperature) are unbounded real numbers. A bounded activation like sigmoid (range 0–1) or tanh (range -1–1) would make it mathematically impossible for the network to ever predict values outside that range, regardless of training.
Practice Question
Explain why a 4-layer network using only linear activations throughout is mathematically equivalent to a network with just 1 linear layer.
