A vector space is a formal way of saying "a set of vectors you can add together and scale, and always land back inside the same set." This sounds abstract, but it's the precise mathematical language behind terms you'll use constantly: feature space, embedding space, latent space.
Definition (Informal but Precise)
A vector space is a set \(V\) of vectors, together with addition and scalar multiplication, such that for any vectors \(\mathbf{u}, \mathbf{v} \in V\) and any scalar \(c\):
- \(\mathbf{u} + \mathbf{v}\) is also in \(V\) (closed under addition)
- \(c\mathbf{v}\) is also in \(V\) (closed under scalar multiplication)
- There's a zero vector \(\mathbf{0}\) such that \(\mathbf{v} + \mathbf{0} = \mathbf{v}\) for all \(\mathbf{v}\)
The most common example in deep learning is \(\mathbb{R}^n\) — all real-valued vectors of length \(n\). Every feature vector, weight vector, and embedding you'll work with lives in some \(\mathbb{R}^n\).
Basis, Span and Dimension
| Term | Meaning |
|---|---|
| Span | All vectors reachable by scaling and adding a given set of vectors together |
| Basis | A minimal set of vectors whose span is the entire space — every vector in the space is a unique combination of basis vectors |
| Dimension | The number of vectors in a basis (e.g. \(\mathbb{R}^n\) has dimension \(n\)) |
| Linear independence | A set of vectors where none can be written as a combination of the others — a basis must be linearly independent |
Numerical Example
In \(\mathbb{R}^2\), the standard basis is \(\mathbf{e}_1 = [1,0]\) and \(\mathbf{e}_2 = [0,1]\). Any vector, e.g. \([3, -2]\), can be written uniquely as \(3\mathbf{e}_1 - 2\mathbf{e}_2\). This is why \(\mathbb{R}^2\) has dimension 2 — exactly two independent directions are needed to reach every point.
Code — Checking Linear Independence
import numpy as np
# Rows of this matrix are the vectors; rank tells you how many are independent
vectors = np.array([[1, 2], [2, 4]]) # second row = 2 * first row
print(np.linalg.matrix_rank(vectors)) # 1 -> NOT independent, span is only a line, not all of R^2
independent_vectors = np.array([[1, 0], [0, 1]])
print(np.linalg.matrix_rank(independent_vectors)) # 2 -> independent, spans all of R^2Where This Shows Up in Deep Learning
- Feature space: a dataset with \(n\) numeric features lives in \(\mathbb{R}^n\) — every model you train is learning a function over this vector space.
- Embedding space: a word embedding maps each word to a point in \(\mathbb{R}^d\) (e.g. \(d=300\)) — a carefully learned vector space where distance and direction encode semantic meaning (see Word Embeddings).
- Latent space: autoencoders and GANs compress data into a lower-dimensional vector space where each dimension (ideally) captures a meaningful factor of variation.
Common Mistakes
- Confusing "vector space" with just "a collection of vectors" — the closure and structure requirements matter; not every arbitrary set of vectors forms a valid vector space on its own operations.
- Assuming a higher-dimensional embedding space is automatically "better" — dimensionality is a design choice with real tradeoffs in compute, overfitting risk, and interpretability.
Interview Relevance
Q: "What does it mean for a set of feature vectors to be linearly dependent, and why does it matter?" It means at least one feature can be written as a combination of the others — it carries no new information. In practice this can cause numerical instability (e.g. in matrix inversion) and is a signal to drop or combine redundant features before training.
Practice Question
Are \([1, 1, 0]\), \([0, 1, 1]\) and \([1, 2, 1]\) linearly independent? (Hint: check whether the third can be written as a combination of the first two.)
