Skip-Gram is Word2Vec's second architecture — the mirror image of CBOW: instead of predicting a target word from its context, it predicts each surrounding context word individually, given the target word.
The Task, Concretely
Given the target word "brown" (from "the quick brown fox jumps," window size 2), skip-gram generates separate training examples predicting each of "the," "quick," "fox," and "jumps" individually — one prediction task per context word, rather than CBOW's single averaged prediction.
Formula
The target word's own embedding \(\mathbf{e}_{w_t}\) is used directly (no averaging) to predict each surrounding context word's probability, one at a time.
Diagram
The target word independently predicts each context word — generating multiple separate training examples per window, unlike CBOW's single averaged one.
CBOW vs Skip-Gram — Direct Comparison
| CBOW | Skip-Gram | |
|---|---|---|
| Predicts | Target word from context | Context words from target |
| Training examples per window | One (averaged) | Multiple (one per context word) |
| Training speed | Faster | Slower |
| Performance on rare words | Weaker — rare words get "averaged out" alongside common ones | Stronger — each rare word gets its own dedicated training examples as a target |
| Best suited to | Larger datasets, faster iteration | Tasks where rare-word quality matters |
Code
from gensim.models import Word2Vec
sentences = [["the", "quick", "brown", "fox", "jumps"],
["the", "lazy", "dog", "sleeps"]]
model = Word2Vec(sentences, vector_size=100, window=2, sg=1) # sg=1 selects skip-gram specifically
print(model.wv.most_similar("fox"))Common Mistakes
- Assuming skip-gram is strictly "better" than CBOW in every case — CBOW's speed advantage and reasonable performance on common words make it a genuinely sensible choice when training data is very large and rare-word quality matters less.
- Confusing the direction of prediction between the two — a quick way to remember: CBOW predicts one word from many (bag of context words); skip-gram predicts many words from one (skips outward from the target).
Interview Relevance
Q: "Why does skip-gram typically produce better embeddings for rare words than CBOW?" In CBOW, a rare word appearing as part of the averaged context gets its individual signal diluted/blended together with the other, often more common, context words in that same window. In skip-gram, a rare word used as the target word directly generates its own dedicated training examples for each surrounding context word, giving it more focused, individual training signal.
Practice Question
For the target word "guitar" in "she plays guitar well," with window size 1, how many separate training examples would skip-gram generate from this one window?
