BERT (Bidirectional Encoder Representations from Transformers) is an encoder-only Transformer, pretrained to produce rich contextual embeddings — and one of the most influential NLP models of the modern deep learning era.
"Bidirectional" — The Key Distinguishing Feature
Recall from Masked Self-Attention that a decoder's self-attention must be masked, seeing only earlier tokens, because generation happens left-to-right. BERT is different: it's built entirely from encoder layers (see Transformer Encoder), whose self-attention is not masked — every token can attend to every other token in both directions, before and after it. This bidirectional context is exactly what makes BERT so effective at understanding tasks, where seeing the full sentence at once (not just what came before a given word) is valuable.
Pretraining Objective 1: Masked Language Modeling (MLM)
Randomly mask out (replace with a special [MASK] token) about 15% of the input tokens, and train the model to predict the original masked tokens from the surrounding (bidirectional) context:
This is a self-supervised task (no human labels needed, exactly like Word2Vec's approach) — but critically, it requires bidirectional context, which is exactly why this specific pretraining task and BERT's bidirectional architecture were designed together.
Pretraining Objective 2: Next Sentence Prediction (NSP)
Given two sentences, predict whether the second genuinely follows the first in the original text, or is a randomly sampled unrelated sentence — intended to help the model learn relationships between sentences, useful for tasks like question answering. (Later research, including RoBERTa in the next note, found this specific objective contributed less than originally believed.)
Special Tokens and Fine-Tuning
BERT's input format uses a [CLS] token at the start (whose final representation is commonly used as a whole-sentence summary for classification tasks) and [SEP] tokens to separate sentence pairs. After pretraining on huge amounts of unlabeled text, BERT is typically fine-tuned (see the Transfer Learning category) on a much smaller labeled dataset for a specific downstream task — classification, named entity recognition, question answering, and more.
Code
from transformers import BertTokenizer, BertForSequenceClassification
import torch
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2)
inputs = tokenizer("This movie was fantastic!", return_tensors="pt")
outputs = model(**inputs)
predicted_class = outputs.logits.argmax(dim=-1)
print(predicted_class) # 0 or 1 -- e.g. negative/positive sentiment, after fine-tuningCommon Mistakes
- Using BERT for text generation — its bidirectional design (seeing the whole sentence at once) is fundamentally mismatched with autoregressive, left-to-right generation; decoder-only models like GPT (next-but-one note) are built specifically for that.
- Forgetting BERT needs task-specific fine-tuning before it's useful for a specific downstream task — the pretrained model alone only knows how to fill in masked words and judge sentence relationships, not classify sentiment or answer questions directly.
Interview Relevance
Q: "Why is BERT well-suited to understanding tasks but not text generation?" BERT's bidirectional self-attention lets every token see the entire sentence, both before and after it, which is exactly what's needed for tasks that benefit from full-sentence context (classification, extraction, question answering). Text generation is inherently left-to-right and autoregressive — a token being generated genuinely cannot see "future" tokens that don't exist yet — which is incompatible with BERT's unmasked, fully bidirectional design.
Practice Question
Why does Masked Language Modeling specifically require a bidirectional architecture, unlike next-token prediction (used by GPT-style models)?
