Before embeddings, the most obvious way to represent a word numerically was one-hot encoding — a vector as long as the entire vocabulary, all zeros except a single 1 marking that specific word. This note explains exactly why this approach was abandoned in favor of dense embeddings.
Formula
\(|V|\) is the vocabulary size. Every word gets its own vector, as long as the entire vocabulary, with a 1 at exactly its own index and 0 everywhere else.
The Two Fatal Problems
| Problem | Detail |
|---|---|
| Extreme dimensionality | For a 50,000-word vocabulary, every single word is a 50,000-dimensional vector — almost entirely zeros, wasting enormous memory and compute |
| No notion of similarity | Any two distinct one-hot vectors have a dot product of exactly 0 — "cat" and "dog" are represented as exactly as "similar" (i.e. not at all) as "cat" and "asteroid" |
Numerical Example — Verifying the Similarity Problem
For a tiny 4-word vocabulary {cat, dog, car, run}: one-hot("cat")\(=[1,0,0,0]\), one-hot("dog")\(=[0,1,0,0]\).
Despite "cat" and "dog" being semantically related (both animals), their one-hot vectors are exactly as dissimilar (dot product 0) as "cat" and "car," or "cat" and "run" — one-hot encoding has no mechanism for encoding this kind of similarity at all, since every pair of distinct words is equally "orthogonal."
Code
import torch
vocab = {"cat": 0, "dog": 1, "car": 2, "run": 3}
vocab_size = len(vocab)
def one_hot(word):
vec = torch.zeros(vocab_size)
vec[vocab[word]] = 1
return vec
cat_vec = one_hot("cat")
dog_vec = one_hot("dog")
print(torch.dot(cat_vec, dog_vec)) # tensor(0.) -- no similarity captured at allWhat This Directly Motivates
Both problems point toward the same fix: replace this huge, sparse, meaningless representation with a much smaller, dense vector — where the actual numeric values are learned such that semantically similar words end up with similar vectors. This is exactly what Word Embeddings, the next note, provides.
Common Mistakes
- Assuming one-hot encoding is purely a historical curiosity with no modern relevance — it's still the conceptual starting point that motivates embeddings, and one-hot representations still appear directly as the target format in classification tasks (e.g. one-hot encoded labels in categorical cross-entropy, from Categorical Cross-Entropy), just not typically as the input representation for words anymore.
Interview Relevance
Q: "What are the two main problems with one-hot encoding words, and how do embeddings solve them?" One-hot vectors are extremely high-dimensional and sparse (one dimension per vocabulary word, mostly zeros), wasting memory and compute; and they have no notion of similarity — every pair of distinct words is equally "orthogonal," regardless of actual meaning. Embeddings solve both by using much lower-dimensional, dense vectors whose values are learned so that semantically similar words end up close together in the embedding space.
Practice Question
For a vocabulary of 100,000 words, how many dimensions would a one-hot vector need? How does that compare to a typical embedding dimension of 300?
