Word2Vec (Mikolov et al., 2013) was the breakthrough that made high-quality, efficiently-trained word embeddings practical and popular — introducing two related training approaches, both covered individually in the next two notes.
The Core Training Idea
Word2Vec learns embeddings not by any explicit definition of "meaning," but by a simple, self-supervised prediction task over huge amounts of raw text: use nearby words to predict each other. No human labeling is required at all — the "labels" are generated automatically from the raw text itself (a word's actual neighboring words), making this a form of self-supervised learning (see Types of Learning).
The Two Architectures — A Preview
| Architecture | Predicts | Covered In |
|---|---|---|
| CBOW (Continuous Bag of Words) | The target word, from its surrounding context words | CBOW |
| Skip-Gram | The surrounding context words, from the target word | Skip-Gram |
Both are trained on the exact same underlying idea — "words in similar contexts have similar meanings" — just with the prediction direction flipped. Both produce, as a side effect of solving their prediction task, a learned embedding matrix that ends up capturing rich semantic structure.
The Embedding Is a Byproduct, Not the Direct Target
This is a subtle but important point: Word2Vec's actual training objective is predicting words from context (or vice versa) — a task nobody actually cares about achieving perfectly for its own sake. What's actually useful is the embedding matrix the network learns along the way, as a necessary intermediate representation for solving that prediction task well. Once training finishes, the prediction task itself is typically discarded, and only the learned embedding matrix is kept and reused.
Code — Using a Pretrained Word2Vec Model
import gensim.downloader as api
model = api.load("word2vec-google-news-300") # pretrained on ~100 billion words
print(model.most_similar("king", topn=3))
# [('kings', 0.71), ('monarch', 0.64), ('queen', 0.63)] -- learned semantic neighbors
print(model.similarity("cat", "dog")) # a high similarity score
print(model.similarity("cat", "car")) # a much lower similarity scoreCommon Mistakes
- Assuming Word2Vec understands word meaning in any deep sense — it purely learns statistical co-occurrence patterns from raw text; it has no grounding in real-world meaning beyond what's implied by word co-occurrence patterns in its training corpus.
- Using Word2Vec embeddings and expecting context-sensitivity — as covered in Contextual Embeddings, Word2Vec produces exactly one fixed vector per word, regardless of which sentence it appears in — "bank" (river) and "bank" (money) get the identical vector.
Interview Relevance
Q: "How does Word2Vec learn word embeddings without any human-labeled data?" It sets up a self-supervised prediction task using the natural structure of text itself — predicting a word from its surrounding context, or vice versa — where the "labels" are simply which words happen to co-occur nearby in a large text corpus. No human annotation is needed; the embedding matrix is learned as a necessary byproduct of the network solving this prediction task well.
Practice Question
Why is Word2Vec considered a form of self-supervised learning rather than supervised or unsupervised learning in the strictest sense?
