PEFT (Parameter-Efficient Fine-Tuning) is the umbrella term for techniques that adapt a large pretrained model to a new task by training only a small number of new or added parameters — freezing the vast majority of the original model entirely.
The Core Idea
Instead of updating all of a model's billions of parameters (full fine-tuning), PEFT methods introduce a small number of new, trainable parameters — often well under 1% of the original model's size — while keeping every original pretrained weight completely frozen, exactly like the frozen-backbone pattern from Freezing Layers, just applied far more aggressively and cleverly.
Why This Works Surprisingly Well
Research has found that adapting a large pretrained model to a new task often doesn't require moving its weights very far from their pretrained values — the "direction" of adaptation needed frequently lies in a much lower-dimensional space than the model's full parameter count would suggest. PEFT techniques are specifically designed to capture exactly this — a small, targeted adjustment — rather than allowing unrestricted movement across every one of billions of parameters.
The Family of Techniques, Previewed
| Technique | Core Idea | Covered In |
|---|---|---|
| LoRA | Add a small, low-rank trainable update alongside frozen weights | LoRA |
| QLoRA | LoRA combined with quantizing the frozen base model | QLoRA |
| Adapters | Small trainable bottleneck modules inserted between frozen layers | Adapters |
| Prefix/Prompt Tuning | Trainable "virtual tokens" prepended to inputs or hidden states | Prefix Tuning, Prompt Tuning |
Code — The General Pattern
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
peft_config = LoraConfig(r=8, lora_alpha=16, target_modules=["q_proj", "v_proj"])
peft_model = get_peft_model(model, peft_config)
peft_model.print_trainable_parameters()
# trainable params: ~4,194,304 || all params: ~6,738,415,616 || trainable%: 0.062%Common Mistakes
- Assuming PEFT always matches full fine-tuning's performance exactly — for many tasks it comes remarkably close, but for tasks requiring substantial adaptation, full fine-tuning can still outperform PEFT, a real tradeoff worth validating empirically.
- Conflating PEFT with a single specific technique — it's a category encompassing several genuinely distinct approaches (LoRA, adapters, prompt/prefix tuning), each with different mechanics and tradeoffs.
Interview Relevance
Q: "Why can training well under 1% of a model's parameters often achieve performance close to full fine-tuning?" Research on large pretrained models suggests that the adaptation needed for many downstream tasks lies in a surprisingly low-dimensional subspace relative to the model's full parameter count — the pretrained weights already encode most of the necessary general knowledge, and only a small, targeted adjustment is typically needed to specialize toward a new task, which PEFT methods are specifically designed to capture efficiently.
Practice Question
What's the practical benefit of PEFT beyond just saving GPU memory during training — think about model storage and deployment for multiple fine-tuned variants of the same base model.
