This note revisits embeddings specifically through the lens of modern AI applications — how dense vector representations of text, images, and other data power the retrieval, search, and agent systems covered throughout this category.
The Core Idea, Recap
An embedding maps a piece of data — a word, sentence, document, or image — into a dense vector of real numbers \(\mathbf{v} \in \mathbb{R}^d\), positioned in a continuous space such that semantically similar inputs end up close together. Modern AI systems rely on this heavily: rather than comparing raw text or pixels directly, they compare embedding vectors, where "similar meaning" becomes "close in vector space" — a computationally tractable operation.
Modern Embedding Models
| Model Family | Typical Use |
|---|---|
| Sentence/text embedding models (e.g. built on Transformer encoders) | Semantic search, document similarity, retrieval for RAG systems |
| CLIP-style multimodal embeddings | Placing images and text into a shared embedding space, enabling cross-modal search |
| Embeddings from LLM hidden states | Repurposing a large language model's internal representations for downstream retrieval tasks |
Code — Generating and Comparing Text Embeddings
from sentence_transformers import SentenceTransformer
import numpy as np
model = SentenceTransformer('all-MiniLM-L6-v2')
sentences = [
"The cat sat on the mat.",
"A feline rested on the rug.",
"The stock market crashed today."
]
embeddings = model.encode(sentences) # shape: (3, 384)
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
print(cosine_similarity(embeddings[0], embeddings[1])) # high -- similar meaning
print(cosine_similarity(embeddings[0], embeddings[2])) # low -- unrelated meaningDespite sharing almost no words in common, the first two sentences produce embeddings with high cosine similarity because they express similar meaning — exactly the property that makes embeddings useful for semantic (meaning-based) search, rather than the brittle keyword matching of traditional search.
Why Embeddings Are the Foundation of Modern AI Retrieval
Every system in this category that needs to "find relevant information" — RAG, semantic search, recommendation — ultimately reduces to computing embeddings and comparing them efficiently, which is exactly why the next note, vector databases, exists: to store and search over millions or billions of these vectors efficiently.
Common Mistakes
- Comparing embeddings from two different, incompatible models — embedding spaces are specific to the model that produced them; vectors from different models aren't meaningfully comparable to each other.
- Using a general-purpose embedding model for a highly specialized domain (e.g. legal or medical text) without evaluating whether a domain-specific or fine-tuned embedding model would perform meaningfully better.
Interview Relevance
Q: "Why can't you directly compare embedding vectors produced by two different embedding models?" Each embedding model learns its own internal geometry during training — the specific dimensions, scale, and relative positioning of its vector space are specific to that model's training process and objective. Two different models can place semantically similar concepts at entirely different, incompatible coordinates in their respective spaces, so a vector from one model has no meaningful relationship to a vector from another — comparisons (like cosine similarity) are only valid between embeddings produced by the same model.
Practice Question
Why does cosine similarity, rather than raw text matching, allow two sentences with completely different wording to be identified as semantically similar?
