Supervised Fine-Tuning (SFT) is the stage right after pretraining: continue training the model, now on a much smaller, carefully curated dataset of high-quality example responses — reshaping the raw, text-continuation model toward producing genuinely useful outputs.
Pretraining vs SFT — Directly Compared
| Pretraining | Supervised Fine-Tuning | |
|---|---|---|
| Data source | Enormous raw text corpora (trillions of tokens) | A much smaller set of curated, high-quality (prompt, response) pairs (thousands to low millions) |
| Data origin | Naturally occurring text — no human curation needed | Often written or carefully selected by humans specifically to demonstrate desired behavior |
| Goal | Learn general language, knowledge, reasoning | Shape the model toward producing helpful, well-formatted, on-task responses |
| Training objective | Next-token prediction on raw text | Same underlying next-token prediction, but now specifically on the target response text |
What SFT Data Actually Looks Like
{
"prompt": "Explain photosynthesis in simple terms.",
"response": "Photosynthesis is how plants make their own food using sunlight..."
}
{
"prompt": "Write a Python function to reverse a string.",
"response": "def reverse_string(s):\n return s[::-1]"
}The model is trained to predict the response text, given the prompt — the exact same next-token prediction mechanism from Next-Token Prediction, just applied to this specifically curated data instead of raw web text.
Why a Much Smaller Dataset Is Enough Here
Pretraining needs to teach the model language and knowledge essentially from scratch, requiring enormous scale. SFT starts from an already highly capable pretrained model — its job is comparatively narrower: teach the model the specific format and style of a helpful response, not new knowledge. This is a much easier learning problem, achievable with orders of magnitude less data.
Code
from transformers import Trainer, TrainingArguments
# Conceptually: SFT is standard supervised fine-tuning, using the SAME
# cross-entropy loss as pretraining, just on the curated (prompt, response) dataset
training_args = TrainingArguments(
output_dir="./sft-output",
num_train_epochs=3,
per_device_train_batch_size=4,
learning_rate=2e-5, # typically much smaller than pretraining's learning rate
)
# trainer = Trainer(model=pretrained_model, args=training_args, train_dataset=sft_dataset)
# trainer.train()Common Mistakes
- Assuming SFT alone is sufficient for a production-quality assistant — SFT shapes format and basic helpfulness, but further alignment stages (RLHF/DPO, covered next) are typically needed to more precisely tune the model's behavior toward nuanced human preferences and safety.
- Using low-quality or inconsistent SFT data — since this stage directly shapes the model's response style and quality, poor examples here can noticeably degrade the model's behavior, even if the underlying pretrained model is strong.
Interview Relevance
Q: "Why does SFT need far less data than pretraining, even though both use the same next-token prediction training mechanism?" Pretraining has to teach a model language, facts, and reasoning essentially from nothing, which requires enormous data volume. SFT starts from an already highly capable pretrained model and only needs to teach it the specific format and style of desired responses — a comparatively narrow adjustment, not building new capability from scratch, so far less data is needed to achieve it.
Practice Question
Why would training only via SFT on a narrow set of curated examples risk making a model perform worse on tasks outside that specific data's style or domain?
