Masked Image Modeling

Masked Image Modeling is the vision counterpart to Masked Language Modeling — mask out random patches of an image, and train a model to reconstruct the missing content, directly translating MLM's proven pretext-task recipe into the visual domain.

The Core Recipe (Following the MAE Approach)

1. Split an image into a grid of fixed-size patches (a pattern shared with Vision Transformers, covered elsewhere).
2. Randomly mask out a large fraction of these patches — commonly as much as 75%, notably far more aggressive than MLM's typical 15%.
3. Feed only the small set of visible patches through an encoder.
4. Use a lightweight decoder to reconstruct the original pixel values of the masked patches, from the encoder's output plus positional information.

Why Such a High Masking Ratio Works for Images

Images have far more spatial redundancy than text — neighboring pixels and patches are often highly correlated (a patch of sky looks similar to the sky patches next to it), so predicting a masked patch is often solvable just from very local context, unless a much larger fraction is masked. This is precisely why effective masked image modeling requires masking a much higher percentage of the image than MLM masks of text — to force the model to rely on genuine, higher-level scene understanding rather than trivial local interpolation.

Diagram

Code

import torch

def mask_patches(patches, mask_ratio=0.75):
    num_patches = patches.shape[1]
    num_masked = int(num_patches * mask_ratio)

    perm = torch.randperm(num_patches)
    masked_indices = perm[:num_masked]
    visible_indices = perm[num_masked:]

    visible_patches = patches[:, visible_indices]   # ONLY these feed the encoder
    return visible_patches, visible_indices, masked_indices

patches = torch.randn(1, 196, 768)   # e.g. a 224x224 image split into 14x14=196 patches
visible, vis_idx, masked_idx = mask_patches(patches, mask_ratio=0.75)
print(visible.shape)   # (1, 49, 768) -- only 25% of patches visible to the encoder

An Efficiency Bonus

Because only the small set of visible patches actually feeds the (typically much larger, more expensive) encoder, this approach — compared to feeding the full image with masked patches simply zeroed out — is also considerably more compute-efficient to pretrain, since the encoder never has to process the masked patches at all.

Common Mistakes

• Using a masking ratio similar to text's typical 15% for images — given images' much greater spatial redundancy, this would make the reconstruction task too easy, solvable via trivial local interpolation rather than genuine scene understanding.
• Feeding masked (zeroed-out) patches through the encoder instead of excluding them entirely — this wastes compute on positions carrying no useful information, forgoing the efficiency benefit of processing only visible patches.

Interview Relevance

Q: "Why does masked image modeling typically use a much higher masking ratio (e.g. 75%) than masked language modeling's typical 15%?" Images have far more spatial redundancy than text — a masked patch is often trivially predictable from immediately neighboring, unmasked patches unless a much larger fraction of the image is masked. A high masking ratio forces the model to rely on genuine, higher-level understanding of the scene's content, rather than simple local interpolation from nearby visible pixels.

Practice Question

Why does feeding only visible patches (rather than the full image with masked patches zeroed) provide a meaningful compute efficiency benefit during pretraining?

Related DL Notes

• Masked Language Modeling
• Autoencoders
• Representation Learning