A practical reference for every activation function covered conceptually in the Activation Functions category, showing both the module and functional forms PyTorch provides.
Module vs Functional Form
import torch
import torch.nn as nn
import torch.nn.functional as F
x = torch.randn(5)
# Module form -- used as a LAYER inside nn.Sequential or __init__
relu_layer = nn.ReLU()
output1 = relu_layer(x)
# Functional form -- called DIRECTLY inside forward(), no separate layer object needed
output2 = F.relu(x)
print(torch.equal(output1, output2)) # True -- identical computation, different calling conventionThe module form is preferred when the activation needs to be part of an nn.Sequential chain or tracked as a submodule; the functional form is common (and slightly more concise) when writing a custom forward() method directly.
Common Activations, Quick Reference
| Activation | Module | Functional | Concept Note |
|---|---|---|---|
| ReLU | nn.ReLU() | F.relu(x) | ReLU |
| Sigmoid | nn.Sigmoid() | torch.sigmoid(x) | Sigmoid Function |
| Tanh | nn.Tanh() | torch.tanh(x) | Tanh Function |
| Softmax | nn.Softmax(dim=-1) | F.softmax(x, dim=-1) | Softmax Function |
| GELU | nn.GELU() | F.gelu(x) | GELU |
| Leaky ReLU | nn.LeakyReLU(0.01) | F.leaky_relu(x, 0.01) | Leaky ReLU |
Common Mistakes
- Forgetting to specify
dimforSoftmax/F.softmax— softmax must normalize along a specific dimension (typically the class dimension); omitting or mis-specifying this produces silently incorrect probabilities. - Applying
nn.Softmaxbeforenn.CrossEntropyLoss, which already applies it internally — this exact double-softmax mistake has been flagged repeatedly across this hub because it's genuinely one of the most common practical PyTorch bugs.
Interview Relevance
Q: "What's the practical difference between using nn.ReLU() and F.relu() in a model's forward pass?" They compute the exact same operation — the difference is purely about code structure. nn.ReLU() creates a reusable layer object, typically assigned in __init__ and useful within nn.Sequential chains or when you need the activation tracked as a named submodule. F.relu() is a direct functional call, often used inline inside a custom forward() method without needing a separate layer instance.
Practice Question
Why must the dim argument for softmax be chosen carefully for a batch of shape (batch_size, num_classes)?
