Adapters take a structurally different approach to PEFT than LoRA: instead of modifying existing weight matrices with a low-rank update, insert small, new bottleneck modules directly between a pretrained model's frozen layers, and train only those.
The Adapter Module
\(\mathbf{W}_{\text{down}}\) projects the input down to a small bottleneck dimension, \(\phi\) applies a non-linearity, and \(\mathbf{W}_{\text{up}}\) projects back up to the original dimension — structurally a tiny autoencoder-like module (recall Autoencoders), inserted with a residual connection (see Residual Connections) so it starts as a near-identity function.
Where Adapters Get Inserted
Typically, one or two small adapter modules are inserted into each Transformer block — after the attention sublayer, and after the feed-forward sublayer — with every other original weight in the model kept completely frozen. Only these newly added adapter parameters (and often the layer normalization parameters) are trained.
Adapters vs LoRA — Direct Comparison
| Adapters | LoRA | |
|---|---|---|
| Mechanism | New modules inserted between layers | Low-rank update added alongside existing weight matrices |
| Adds inference latency? | Yes — extra sequential computation at inference time | No (if merged) — \(\mathbf{B}\mathbf{A}\) can be added directly into \(\mathbf{W}\) after training, adding zero extra inference cost |
| Structural change | New layers in the computation graph | No new layers — modifies existing matrix multiplications |
This latency difference is a genuinely important practical distinction — it's a large part of why LoRA became more popular than classic adapters for latency-sensitive LLM deployment specifically.
Code
import torch
import torch.nn as nn
class Adapter(nn.Module):
def __init__(self, d_model, bottleneck_dim=64):
super().__init__()
self.down_proj = nn.Linear(d_model, bottleneck_dim)
self.up_proj = nn.Linear(bottleneck_dim, d_model)
self.activation = nn.GELU()
nn.init.zeros_(self.up_proj.weight) # start as a near-identity function
def forward(self, x):
return x + self.up_proj(self.activation(self.down_proj(x))) # residual: near-zero change initially
adapter = Adapter(d_model=768, bottleneck_dim=64)
trainable_params = sum(p.numel() for p in adapter.parameters())
print(trainable_params) # a tiny fraction of a full Transformer layer's parameter countCommon Mistakes
- Assuming adapters add zero inference cost the way merged LoRA weights can — adapters remain genuinely separate layers at inference time, adding real (if typically small) sequential computation.
- Choosing an adapter bottleneck dimension without considering the same low-rank/capacity tradeoff that applies to LoRA's rank \(r\) — too small a bottleneck limits what the adapter can learn to adjust.
Interview Relevance
Q: "Why might LoRA be preferred over classic adapters for latency-sensitive LLM deployment?" LoRA's low-rank update \(\mathbf{B}\mathbf{A}\) can be merged directly into the original frozen weight matrix after training (\(\mathbf{W}'=\mathbf{W}+\mathbf{B}\mathbf{A}\)), adding zero extra computation at inference time. Adapters remain genuinely separate, sequentially-executed modules inserted into the model's computation graph, adding real (if small) additional inference latency that can't be eliminated the same way.
Practice Question
Why is the adapter module's up-projection weight initialized to zero, similar to LoRA's \(\mathbf{B}\) matrix?
