Image captioning generates a natural-language sentence describing an image's content — combining a CNN's visual understanding with a sequence-generating decoder, in exactly the encoder-decoder pattern that will get its own full treatment in the Sequence-to-Sequence category.
The Encoder-Decoder Pipeline
| Component | Role |
|---|---|
| CNN encoder | Processes the image into a compact feature representation (often the output right before a classification CNN's final layer, or a set of spatial feature maps) |
| RNN or Transformer decoder | Generates the caption one word at a time, conditioned on the image's encoded features and the words generated so far |
Why This Is a Cross-Modal Task
Image captioning uniquely bridges two very different data modalities within a single model — visual features (from the CNN encoder) and natural language (from the decoder) — an early, foundational example of the kind of multimodal learning covered more fully in the Advanced Deep Learning category, and a direct conceptual ancestor of modern vision-language models.
Attention Improves Captioning Significantly
Rather than compressing an entire image into one fixed vector (losing fine spatial detail), attention-based captioning models (previewing the Attention category) let the decoder "look at" different specific regions of the image while generating each word — e.g. focusing on a dog's region specifically while generating the word "dog." This significantly improves caption quality and specificity over a purely fixed-vector encoding approach.
Code — A Simplified Architecture Sketch
import torch
import torch.nn as nn
import torchvision.models as models
class ImageCaptioningModel(nn.Module):
def __init__(self, vocab_size, embed_size=256, hidden_size=512):
super().__init__()
cnn = models.resnet18(weights='IMAGENET1K_V1')
self.encoder = nn.Sequential(*list(cnn.children())[:-1]) # CNN feature extractor
self.embed = nn.Linear(512, embed_size)
self.decoder = nn.LSTM(embed_size, hidden_size, batch_first=True)
self.fc = nn.Linear(hidden_size, vocab_size)
def forward(self, images):
features = self.encoder(images).flatten(1)
features = self.embed(features).unsqueeze(1)
output, _ = self.decoder(features)
return self.fc(output)Common Mistakes
- Assuming a single fixed image-feature vector is sufficient for high-quality captioning — this loses spatial detail that attention mechanisms specifically recover, generally producing more accurate, specific captions.
- Evaluating generated captions with exact-match accuracy — as with translation and summarization (see BLEU Score and ROUGE Score), captions have many valid phrasings; overlap-based metrics are the standard, more appropriate evaluation approach.
Interview Relevance
Q: "Why does image captioning benefit significantly from an attention mechanism?" Without attention, the entire image must be compressed into one fixed-size vector before the decoder generates any words — losing fine spatial detail relevant to specific words. Attention lets the decoder dynamically focus on different image regions while generating each word, producing more accurate and specific captions that correctly ground individual words in the relevant part of the image.
Practice Question
Why is image captioning considered a cross-modal (multimodal) task, and what are its two distinct data modalities?
