This note goes deeper into parameter count specifically — how it's actually computed for common layer types, and why it's such a commonly reported, if imperfect, measure of model size in research papers.
Computing Parameter Count for Common Layers
| Layer Type | Parameter Count Formula |
|---|---|
| Fully-connected (Linear) | \((\text{input\_dim} \times \text{output\_dim}) + \text{output\_dim}\) — weights plus one bias per output unit |
| Convolutional | \((\text{kernel\_h} \times \text{kernel\_w} \times \text{in\_channels} \times \text{out\_channels}) + \text{out\_channels}\) |
| Embedding layer | \(\text{vocab\_size} \times \text{embedding\_dim}\) |
Code — Counting a Model's Total Parameters in PyTorch
import torch.nn as nn
def count_parameters(model, trainable_only=True):
if trainable_only:
return sum(p.numel() for p in model.parameters() if p.requires_grad)
return sum(p.numel() for p in model.parameters())
model = nn.Sequential(
nn.Linear(784, 256), # (784*256) + 256 = 200,960 parameters
nn.ReLU(),
nn.Linear(256, 10) # (256*10) + 10 = 2,570 parameters
)
total = count_parameters(model)
print(f"Total trainable parameters: {total:,}") # 203,530
# For large models, reporting in millions/billions is standard
print(f"That's {total / 1e6:.2f}M parameters")Why Parameter Count Is So Commonly Reported
It's simple to compute, easy to compare across papers, and correlates loosely with both model capacity and (roughly) compute/memory requirements — making it a convenient, if imperfect, shorthand for describing model scale in research communication. Most papers report parameter count prominently, and it's often used to name model size variants (e.g. "the 7B model," "the 70B model," referring to billions of parameters).
Why It's an Imperfect, Incomplete Proxy
As established in Model Complexity, effective capacity isn't determined by parameter count alone — architecture matters significantly. Additionally, parameter count alone doesn't directly indicate inference compute cost — a Mixture of Experts model (Mixture of Experts) can have an enormous total parameter count while activating only a small fraction per token, meaning its actual inference cost is far lower than its parameter count alone would suggest. This is precisely why FLOPs (the next note) is often reported alongside parameter count, as a complementary, more directly compute-relevant measure.
Common Mistakes
- Assuming parameter count directly and proportionally predicts inference compute cost or latency — this breaks down notably for architectures like Mixture of Experts, where total parameters and per-token active computation can differ dramatically.
- Comparing "model size" purely by parameter count without considering that different architectures use parameters with very different efficiency (e.g. convolutional weight sharing vs fully-connected layers).
Interview Relevance
Q: "Why doesn't a model's total parameter count directly tell you its inference compute cost, especially for architectures like Mixture of Experts?" Parameter count measures total learnable capacity, but inference compute cost depends on how many of those parameters are actually used to process each input. In a standard dense architecture, all parameters participate in every forward pass, so parameter count and compute cost are roughly proportional. In a Mixture of Experts architecture, however, only a small subset of the total experts (and thus a small fraction of total parameters) are activated per token — meaning a model can have an enormous total parameter count while its actual per-token inference compute remains comparable to a much smaller dense model, which is exactly why FLOPs is reported as a separate, complementary measure alongside parameter count.
Practice Question
Why do fully-connected layers typically account for a disproportionately large share of a CNN's total parameter count compared to its convolutional layers?
