Matrix multiplication is the single most-executed operation in deep learning — every linear layer, every attention score, every convolution (reshaped) reduces to it. Understanding its shape rule is non-negotiable for debugging real models.
Formula and Shape Rule
If \(\mathbf{A}\) has shape \((m, n)\) and \(\mathbf{B}\) has shape \((n, p)\), the inner dimensions must match (\(n = n\)), and the result \(\mathbf{C}\) has shape \((m, p)\) — the outer dimensions.
How Each Entry Is Computed
Each output entry \(C_{ij}\) is the dot product of row \(i\) of \(\mathbf{A}\) and column \(j\) of \(\mathbf{B}\).
Full Numerical Example
Matrix Multiplication Is Not Commutative
In general \(\mathbf{A}\mathbf{B} \ne \mathbf{B}\mathbf{A}\) — order matters, unlike scalar multiplication. This is also distinct from element-wise (Hadamard) multiplication, written \(\mathbf{A} \odot \mathbf{B}\), which multiplies corresponding entries and requires equal shapes.
Code
import numpy as np
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])
print(A @ B) # matrix multiplication -> [[19 22] [43 50]]
print(A * B) # element-wise (Hadamard) -> [[5 12] [21 32]] -- different!import torch
A = torch.tensor([[1., 2.], [3., 4.]])
B = torch.tensor([[5., 6.], [7., 8.]])
print(torch.matmul(A, B)) # or A @ B
print(A * B) # element-wise, NOT the same operationWhere This Shows Up in Deep Learning
Every linear (fully-connected) layer computes \(\mathbf{y} = \mathbf{W}\mathbf{x} + \mathbf{b}\) — a matrix multiplication plus a bias. A batch of \(N\) inputs, each of size \(d_{in}\), passed through a layer producing \(d_{out}\) outputs, is one matrix multiplication: \((N, d_{in}) \times (d_{in}, d_{out}) \rightarrow (N, d_{out})\). This same operation, at massive scale, is also the core of self-attention (\(\mathbf{Q}\mathbf{K}^\top\)) covered later in this hub.
Common Mistakes
- Confusing
*(element-wise) with@/matmul(true matrix multiplication) — this is one of the most common silent shape/logic bugs in PyTorch code. - Forgetting the inner-dimension rule and getting a shape-mismatch error without knowing why — always write out both shapes and check the inner numbers match.
Interview Relevance
Q: "What shape does a batch of 32 samples, each with 10 features, become after a linear layer with 5 output units?" The input is \((32, 10)\), the weight matrix is \((10, 5)\), so the matrix multiplication produces \((32, 5)\) — 32 samples, each now with 5 output features.
Practice Question
Can you multiply a \((4, 3)\) matrix by a \((4, 3)\) matrix using standard matrix multiplication? If not, what shape would the second matrix need to be, and what would the result's shape be?
