Instance Segmentation

Instance segmentation adds exactly what semantic segmentation lacks: distinguishing separate objects of the same class, giving each individual object its own unique pixel mask.

Numerical Example — Same Image, Instance-Level

Using the same tiny 4×4 image from Semantic Segmentation, now with instance IDs (0=sky, 1=car instance A, 2=car instance B):

Now the two cars are distinguishable — instance 1 and instance 2 — even though both would share the same semantic class "car." This is a meaningfully richer output than semantic segmentation's single shared label.

How Instance Segmentation Is Typically Built

Rather than a purely pixel-level architecture, instance segmentation commonly combines object detection (to first locate individual object instances via bounding boxes) with a per-instance pixel mask prediction step — exactly the approach taken by Mask R-CNN, covered at the end of this category, which extends Faster R-CNN with an added mask-prediction branch.

Code

import torchvision.models.detection as detection_models

model = detection_models.maskrcnn_resnet50_fpn(weights='DEFAULT')
model.eval()

x = [torch.randn(3, 480, 640)]
predictions = model(x)
print(predictions[0].keys())   # includes 'boxes', 'labels', 'scores', AND 'masks'
print(predictions[0]['masks'].shape)   # one mask per detected instance

Common Mistakes

• Assuming instance segmentation is "just semantic segmentation plus counting" — architecturally, it typically requires a fundamentally different pipeline (detection-then-mask, as in Mask R-CNN) rather than a simple post-processing step on top of semantic segmentation output.

Interview Relevance

Q: "How does instance segmentation typically get built, architecturally?" A common and effective approach (Mask R-CNN) extends an object detector: first detect individual object instances via bounding boxes (as in Faster R-CNN), then predict a precise pixel-level mask within each detected box separately — combining detection's instance-awareness with segmentation's pixel-level precision.

Practice Question

Why is instance segmentation generally considered a harder task than either object detection or semantic segmentation alone?

Related DL Notes

• Semantic Segmentation
• Mask R-CNN
• Panoptic Segmentation