Faster R-CNN (2015) eliminates the last external, non-learned bottleneck in the R-CNN lineage: the region proposal step itself becomes a small neural network, trained jointly with everything else — making the entire detector end-to-end learnable.
Key Innovation: The Region Proposal Network (RPN)
Instead of an external algorithm like Selective Search, Faster R-CNN adds a small, additional neural network — the Region Proposal Network (RPN) — that operates directly on the same shared CNN feature map from Fast R-CNN, predicting candidate object regions itself. Because the RPN is a genuine neural network, it's trained via backpropagation alongside the rest of the detector, rather than being a fixed, separately-engineered algorithm.
Anchor Boxes
The RPN works by proposing adjustments to a set of predefined anchor boxes — a fixed grid of candidate boxes at multiple scales and aspect ratios, placed uniformly across the feature map. For each anchor, the RPN predicts (a) how likely it is to contain an object at all, and (b) how to adjust its exact coordinates to better fit a real object — turning region proposal into a learnable prediction task rather than a fixed, hand-designed algorithm.
The Complete Pipeline
Every stage — feature extraction, region proposal, and classification — is now a neural network, trained jointly end to end.
Advantages and Limitations
| Advantages | Limitations |
|---|---|
| Fully end-to-end trainable — no external, non-learned components | Still a two-stage pipeline (proposal, then classification), inherently more compute-intensive than one-stage detectors |
| Faster and more accurate than both R-CNN and Fast R-CNN | More complex to implement and tune than a single-network one-stage detector |
Use Cases
Faster R-CNN (and its extension, Mask R-CNN, covered at the end of this category) remains a strong, widely-used choice today whenever detection accuracy is prioritized over raw inference speed.
Common Mistakes
- Confusing the Region Proposal Network with the final classification head — they're two distinct neural network components within Faster R-CNN, trained together but serving different roles (proposing candidate regions vs. classifying/refining them).
Interview Relevance
Q: "What does the Region Proposal Network in Faster R-CNN actually replace, and why is that significant?" It replaces the external, non-learned region proposal algorithm (like Selective Search) used in R-CNN and Fast R-CNN with a small neural network that predicts candidate regions directly from the shared CNN feature map. Because it's a genuine neural network, it can be trained jointly with the rest of the detector via backpropagation, making the entire pipeline — from raw image to final detections — end-to-end learnable.
Practice Question
What are anchor boxes, and what two things does the RPN predict for each one?
