Prefix tuning takes yet another distinct approach: instead of modifying any existing weights at all, prepend a small number of trainable "virtual token" vectors directly to the key and value sequences at every attention layer — steering the frozen model's behavior purely through this additional context.
The Core Idea
Recall from Query, Key, Value that attention operates over a sequence of keys and values. Prefix tuning prepends a fixed number \(L\) of learnable vectors to the key and value sequences at every Transformer layer — these "virtual tokens" have no corresponding real input words; they exist purely as trainable parameters that every real token's query can attend to.
Formula, Conceptually
\(\mathbf{P}_K\) and \(\mathbf{P}_V\) are the trainable prefix key/value vectors, prepended to the real sequence's actual keys and values \(\mathbf{K}, \mathbf{V}\). The rest of the model — every weight matrix, every layer — remains completely frozen; only these prefix vectors (a separate, small set per layer) are trained.
Why This Can Work — Steering Through "Soft Context"
Every real token's self-attention now has access to these additional, trainable key/value pairs, and can learn to attend to them when useful — effectively, the prefix vectors act as a task-specific "soft context" the frozen model can consult, without any of the model's own weights ever changing. This is conceptually a more sophisticated, learnable relative of simply prepending a fixed instruction to a prompt.
Code — Using PEFT's Built-in Support
from peft import PrefixTuningConfig, get_peft_model
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("gpt2")
prefix_config = PrefixTuningConfig(
task_type="CAUSAL_LM",
num_virtual_tokens=20 # 20 trainable prefix vectors added at every layer
)
peft_model = get_peft_model(model, prefix_config)
peft_model.print_trainable_parameters() # a tiny fraction of the base model's parametersCommon Mistakes
- Confusing prefix tuning's trainable, layer-by-layer vectors with simply typing extra words into a text prompt — prefix tuning's prefixes are continuous, learned vectors that don't correspond to any actual vocabulary tokens, injected directly at the attention level, not natural language text.
- Using too few or too many virtual tokens without tuning this as a real hyperparameter — the appropriate number, like LoRA's rank, involves a capacity/efficiency tradeoff specific to the task.
Interview Relevance
Q: "How does prefix tuning adapt a frozen model's behavior without changing any of its weights?" It introduces a small number of trainable "virtual token" vectors, prepended to the key and value sequences at every attention layer. These act as additional, learnable context every real token's attention can draw on — steering the frozen model's behavior toward the target task purely through this added context, with every original model weight remaining completely unchanged.
Practice Question
Why must prefix tuning add trainable vectors at every layer, rather than just at the input embedding layer, to effectively influence a deep model's behavior?
