The inverse of a square matrix \(\mathbf{A}\), written \(\mathbf{A}^{-1}\), is the matrix that "undoes" it: multiplying them together produces the identity matrix. Deep learning rarely computes inverses directly (they're expensive and often unstable) — but understanding them is essential for the math underneath classical regression and for reasoning about why iterative optimization exists at all.
Definition
\(\mathbf{I}\) is the identity matrix (1s on the diagonal, 0s elsewhere) — the matrix equivalent of the number 1. Only square matrices can have an inverse, and even then, only if the matrix is non-singular (its determinant is non-zero — see Determinant).
2×2 Inverse Formula
The term \(ad - bc\) is the determinant. If it's zero, the formula divides by zero — the matrix has no inverse (it's singular).
Numerical Example
Code
import numpy as np
A = np.array([[4., 7.], [2., 6.]])
A_inv = np.linalg.inv(A)
print(A_inv)
print(A @ A_inv) # should be (approximately) the identity matrixWhy Deep Learning Avoids Explicit Matrix Inversion
Classical linear regression has a closed-form solution using a matrix inverse, the "normal equation": \(\mathbf{w} = (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y}\). Computing a matrix inverse costs roughly \(O(n^3)\) time — for a weight matrix with millions of parameters, this is completely impractical. This is a core reason deep learning uses iterative optimization (gradient descent and its variants) instead of solving for weights in one closed-form step — see Gradient Descent.
Common Mistakes
- Assuming every matrix has an inverse — only square, non-singular matrices do. A matrix with linearly dependent rows/columns (see Vector Spaces) has determinant 0 and no inverse.
- Using matrix inversion to "solve" large linear systems in production code instead of specialized, numerically stable solvers (e.g.
np.linalg.solve), which are faster and avoid some numerical instability.
Interview Relevance
Q: "Why doesn't deep learning just solve for the optimal weights directly, the way linear regression's normal equation does?" The normal equation requires inverting a matrix whose size scales with the number of parameters — at \(O(n^3)\) cost, this is computationally infeasible for networks with millions/billions of parameters. Gradient-based optimization scales far better and works for the non-linear, non-convex loss surfaces neural networks actually have (where no closed-form solution exists anyway).
Practice Question
Compute the determinant of \(\begin{bmatrix}2 & 4\\1 & 2\end{bmatrix}\). Does this matrix have an inverse? Explain what that means geometrically.
