Alignment is the broad goal underlying the final stages of LLM training: shaping a model's behavior to genuinely match human values, preferences, and safety requirements — going beyond simply performing tasks correctly, toward performing them in ways humans actually want.
Why Alignment Is a Distinct Concern From Capability
A highly capable model (strong at pretraining and SFT) isn't automatically a well-behaved one. A model could be extremely capable at generating fluent, technically correct text while still producing unhelpful, evasive, overly verbose, biased, or unsafe responses — capability and alignment are genuinely separate axes, and a model can score highly on one while falling short on the other.
What Alignment Aims to Achieve
| Goal | What It Means in Practice |
|---|---|
| Helpfulness | Actually addressing what the user asked, in a genuinely useful way |
| Honesty | Not confidently stating incorrect information, expressing appropriate uncertainty |
| Harmlessness | Avoiding harmful, dangerous, or clearly inappropriate outputs |
| Following human preferences | Matching the tone, format, and level of detail humans actually prefer, not just technically satisfying the letter of a request |
How Alignment Is Actually Achieved — A Preview
SFT (previous notes) already provides a first, coarse pass at alignment, by training on examples humans consider good responses. But SFT alone has a real limitation: it only teaches the model to imitate specific example responses — it has no direct signal about degrees of preference (this response is somewhat better than that one) or about avoiding failure modes that weren't explicitly present in the curated examples. The next two notes cover the two dominant techniques specifically designed to incorporate richer, comparative human preference signal: RLHF and DPO.
Common Mistakes
- Treating "alignment" as a solved, one-time step rather than an ongoing area of active engineering and research — different techniques (SFT, RLHF, DPO, and others) each address it partially and imperfectly, and alignment quality is generally evaluated continuously, not verified once and considered permanently done.
- Conflating "aligned" with "harmless only" — alignment explicitly includes helpfulness and honesty as core goals too, not just avoiding harmful outputs; an overly cautious, unhelpful model is also a form of alignment failure.
Interview Relevance
Q: "Why isn't a highly capable pretrained-and-fine-tuned LLM automatically 'aligned' with human preferences?" Capability (fluency, correctness, task performance) and alignment (matching human preferences about helpfulness, honesty, tone, and safety) are distinct properties. A model can be technically very capable while still producing responses humans find unhelpful, evasive, or inappropriate — SFT alone only teaches imitation of specific example responses, without a direct mechanism for learning more nuanced, comparative human preferences, which is exactly what RLHF and DPO are designed to add.
Practice Question
Give an example of a response that could be technically correct and fluent, yet still poorly "aligned" with what a human user actually wanted.
