Swish

Swish was discovered through automated search over candidate activation functions (Google Brain, 2017) rather than hand-designed — and the winning formula turned out to be a remarkably simple one: the input multiplied by its own sigmoid.

Formula

\(\sigma\) is the sigmoid function (see Sigmoid Function). \(\beta\) is either a fixed constant (commonly 1, in which case Swish is also called SiLU, Sigmoid Linear Unit) or, in some variants, a learnable parameter similar to PReLU's approach.

Graph

Swish vs GELU — Close Cousins

Swish (with \(\beta=1\)) and GELU are both smooth, non-monotonic, ReLU-like activations discovered/derived around the same period, and their graphs are visually almost indistinguishable. The key difference is theoretical origin: GELU comes from a probabilistic derivation (weighting by the normal CDF), while Swish emerged from empirical search over a space of candidate functions. In practice, they often perform comparably, and the choice between them is frequently more about which a given published architecture happened to standardize on than a decisive performance gap.

Code

import torch.nn as nn
import torch

layer = nn.SiLU()   # PyTorch's name for Swish with beta=1
x = torch.tensor([-2.0, 0.0, 2.0])
print(layer(x))

# Manual implementation, matching the formula directly
def swish(z, beta=1.0):
    return z * torch.sigmoid(beta * z)
print(swish(x))

Where It's Used Today

Swish (as SiLU) is used in several efficient CNN architectures (including EfficientNet, covered in the CNN Architectures category) and in some diffusion model U-Net implementations. Like GELU, it's smooth and non-monotonic, trading a bit of compute cost for empirically improved optimization in specific architectures.

Common Mistakes

• Treating Swish and GELU as interchangeable without checking which a specific pretrained model actually used — swapping activation functions in a pretrained model (rather than one you're training from scratch) can degrade performance, since the weights were learned assuming a specific activation shape.

Interview Relevance

Q: "How was Swish discovered, and how does that differ from how ReLU or GELU were developed?" Swish was found via automated neural architecture search over a space of candidate activation functions (Google Brain, 2017), rather than derived from a hand-crafted mathematical motivation. ReLU was hand-designed for computational simplicity and non-saturation; GELU was derived from a probabilistic interpretation. Despite the different origins, Swish and GELU end up structurally and empirically very similar.

Practice Question

Using the formula \(\text{Swish}(z)=z\cdot\sigma(\beta z)\), what does Swish reduce to as \(\beta \to \infty\)? (Hint: consider what \(\sigma(\beta z)\) approaches for very large \(\beta\), for \(z>0\) vs \(z<0\).)

Related DL Notes

• GELU
• Sigmoid Function
• EfficientNet