Matrix Addition

Matrix addition combines two matrices of the same shape by adding corresponding elements. It's the simplest matrix operation — and it's the exact mechanism behind adding a bias term and behind residual (skip) connections in modern architectures.

Formula

Each entry of the result is simply the sum of the corresponding entries — this is called an element-wise operation. \(\mathbf{A}\) and \(\mathbf{B}\) must have identical shapes (unless one is broadcastable — see Broadcasting).

Numerical Example

Code

import numpy as np
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])
print(A + B)
# [[ 6  8]
#  [10 12]]

import torch
A = torch.tensor([[1., 2.], [3., 4.]])
B = torch.tensor([[5., 6.], [7., 8.]])
print(A + B)

Where This Shows Up in Deep Learning

• Bias addition: every linear layer computes \(\mathbf{W}\mathbf{x} + \mathbf{b}\) — the \(+\mathbf{b}\) is matrix/vector addition (broadcast across the batch).
• Residual connections: ResNet's core idea is \(\text{output} = \mathbf{x} + F(\mathbf{x})\) — the input is added directly to the transformed output, letting gradients flow through the addition unimpeded. This single operation is central to training very deep networks (see ResNet).

Common Mistakes

• Trying to add matrices of incompatible shapes and expecting an error to explain itself clearly — always check .shape first when addition fails.
• Confusing matrix addition with matrix multiplication — addition is element-wise and requires equal shapes; multiplication has completely different shape rules (see Matrix Multiplication).

Interview Relevance

Q: "What role does matrix addition play in a residual connection?" A residual block computes \(\mathbf{x} + F(\mathbf{x})\) — the original input is added directly to the block's output. Because addition passes gradients through unchanged (derivative of a sum is 1), this addition is what allows gradients to flow to early layers in very deep networks, addressing vanishing gradients.

Practice Question

Given \(\mathbf{A} = \begin{bmatrix}2 & -1\\0 & 3\end{bmatrix}\) and \(\mathbf{B} = \begin{bmatrix}-2 & 1\\4 & 4\end{bmatrix}\), compute \(\mathbf{A} + \mathbf{B}\) by hand, then verify with code.

Related DL Notes

• Matrix Multiplication
• Broadcasting
• Residual Connections