AI Embeddings Explained: Foundations, Fundamentals, and When to Use Them

Introduction

Almost every modern LLM application - semantic search, retrieval-augmented generation, recommendation, deduplication, anomaly detection over text - rests on one underlying idea: turning a piece of content into a list of numbers such that pieces with similar meaning end up with similar numbers. That idea is an embedding, and it's easy to use one without ever understanding what it actually represents, which is exactly how teams end up with retrieval systems that behave mysteriously - surfacing irrelevant results, missing obvious matches, or degrading in ways nobody can explain from the outside.

This article works through embeddings from first principles: what a vector embedding actually encodes, how similarity between vectors is measured and why that measurement is meaningful, how embedding models are trained and what that training implies for how you should use them, and the concrete engineering decisions - chunking, dimensionality, model choice, indexing strategy - that determine whether an embedding-based system actually performs well in production. The goal is to leave you able to reason about why an embedding-based feature is or isn't working, not just how to call an embeddings API.

Context: The Problem Embeddings Solve

Traditional keyword search - the kind built on inverted indexes and term-frequency scoring, as in classic information-retrieval systems like Elasticsearch's BM25 ranking - matches documents based on shared vocabulary. It works well when a query's words are the same words used in the relevant document, and it fails in a specific, predictable way when they aren't: a search for "car won't start" misses a document that says "vehicle fails to turn over," even though the two phrases mean essentially the same thing. This gap between lexical overlap and semantic similarity is the exact problem embeddings exist to close, by representing meaning rather than exact wording.

An embedding model solves this by mapping any piece of text (or, for multimodal models, an image or audio clip) into a fixed-length vector of floating-point numbers - typically somewhere between a few hundred and a few thousand dimensions - positioned in a high-dimensional space such that semantically related inputs land close together and unrelated inputs land far apart. "Car won't start" and "vehicle fails to turn over" end up as two nearby points in this space, despite sharing almost no vocabulary, because the model was trained to capture meaning rather than surface form. This is the property that makes embeddings useful for far more than search: any task that can be reframed as "find things that are similar in meaning" - deduplication, clustering support tickets by topic, recommending related articles, detecting near-duplicate content - can be built on top of the same underlying representation.

The practical reason this has become foundational infrastructure rather than a niche technique is that it decouples "understanding meaning" from "searching efficiently." Computing semantic similarity by asking an LLM to compare every pair of documents directly would be accurate but prohibitively slow and expensive at any real scale. Embeddings solve this by doing the expensive semantic reasoning once, at indexing time, compressing it into a vector, and reducing the runtime comparison to simple, extremely fast vector arithmetic. This is precisely what makes embedding-based retrieval viable for RAG pipelines that need to search millions of documents in milliseconds: the hard part happened in advance, and what's left at query time is cheap.

Deep Technical Explanation: Vector Space, Similarity, and Training

Once text is embedded as a vector, comparing two pieces of content becomes a geometry problem: how close are their two points in the embedding space? The most common metric is cosine similarity, which measures the angle between two vectors rather than the distance between their endpoints, producing a score from -1 (opposite meaning) to 1 (identical meaning) that is insensitive to vector magnitude - useful because embedding magnitude often reflects incidental factors like text length rather than meaning itself. Euclidean distance and dot product are the other two metrics commonly supported by vector databases; dot product is mathematically related to cosine similarity but retains magnitude sensitivity, which matters for some models that are specifically trained assuming dot-product retrieval, while cosine similarity remains the more broadly interoperable default across models.

Embedding models themselves are typically trained using a contrastive learning objective: the model is shown pairs or triplets of examples - an anchor, a genuinely similar example (a positive), and a dissimilar example (a negative), these are adjusted so that its output vectors place the anchor and positive close together while pushing the anchor and negative apart.

This is why the specific data a model was trained on matters enormously for how well it performs on your task: a model trained primarily on general web text and question-answer pairs will represent everyday semantic similarity well, but may perform noticeably worse on a specialized domain - legal contract language, medical terminology, source code - where the notion of "similar meaning" depends on domain-specific structure the training data never emphasized. This is also the reasoning behind the industry's standard benchmark for comparing embedding models, the Massive Text Embedding Benchmark (MTEB), which evaluates models across a broad range of tasks (retrieval, classification, clustering, semantic similarity) precisely because a model that excels at one of these does not automatically excel at the others, and a single aggregate score can obscure which specific capability a given application actually needs.

Dimensionality is the other structural property worth understanding directly rather than treating as an arbitrary configuration knob. Higher-dimensional embeddings can, in principle, encode more nuance, but they also cost more to store and search, and beyond a certain point additional dimensions yield diminishing returns for a given task. A technique called Matryoshka Representation Learning (MRL) - introduced specifically to address this trade-off - trains a single embedding model so that its vector remains meaningful even when truncated to a shorter prefix, letting an application use a smaller, cheaper embedding for coarse filtering and the full vector only where precision actually matters, without needing to train or maintain two separate models.

Implementation: Generating, Comparing, and Indexing Embeddings

Generating an embedding and comparing two vectors directly is worth seeing in raw form before relying on a vector database to hide the mechanics, since understanding this baseline makes it much easier to reason about what any higher-level tool is actually doing underneath. The Python example below computes embeddings for a small set of documents and a query, then ranks the documents by cosine similarity manually.

# embedding_similarity.py
import numpy as np
from openai import OpenAI

client = OpenAI()

def embed(texts: list[str], model: str = "text-embedding-3-small") -> np.ndarray:
    response = client.embeddings.create(input=texts, model=model)
    return np.array([item.embedding for item in response.data])

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> np.ndarray:
    a_norm = a / np.linalg.norm(a, axis=1, keepdims=True)
    b_norm = b / np.linalg.norm(b, axis=1, keepdims=True)
    return a_norm @ b_norm.T

documents = [
    "The vehicle fails to turn over in cold weather.",
    "Our refund policy allows returns within 30 days.",
    "Restart the engine after checking the battery connection.",
]
query = "My car won't start"

doc_embeddings = embed(documents)
query_embedding = embed([query])

scores = cosine_similarity(query_embedding, doc_embeddings)[0]
ranked = sorted(zip(documents, scores), key=lambda x: x[1], reverse=True)

for doc, score in ranked:
    print(f"{score:.3f}  {doc}")

At any meaningful scale, comparing a query against every document with a manual loop like this becomes too slow, since similarity search this way is O(n) in the number of documents. Production systems instead use an Approximate Nearest Neighbor (ANN) index - data structures like HNSW (Hierarchical Navigable Small World graphs) or IVF (Inverted File index) that trade a small amount of retrieval accuracy for dramatically faster lookups, implemented by vector databases such as Pinecone, Weaviate, Qdrant, and pgvector (a PostgreSQL extension), among others.

The TypeScript example below shows a realistic indexing and query pattern against a vector database, including metadata filtering - combining semantic similarity with exact structured constraints, which most real applications need alongside pure vector search.

// vectorSearch.ts
import { QdrantClient } from "@qdrant/js-client-rest";
import OpenAI from "openai";

const openai = new OpenAI();
const qdrant = new QdrantClient({ url: "http://localhost:6333" });

const COLLECTION = "support_articles";

async function embedText(text: string): Promise<number[]> {
  const response = await openai.embeddings.create({
    input: text,
    model: "text-embedding-3-small",
  });
  return response.data[0].embedding;
}

async function indexArticle(
  id: string,
  text: string,
  metadata: { category: string; updatedAt: string }
) {
  const vector = await embedText(text);
  await qdrant.upsert(COLLECTION, {
    points: [{ id, vector, payload: { text, ...metadata } }],
  });
}

async function searchArticles(
  query: string,
  category: string,
  limit = 5
) {
  const queryVector = await embedText(query);

  return qdrant.search(COLLECTION, {
    vector: queryVector,
    limit,
    filter: {
      must: [{ key: "category", match: { value: category } }],
    },
  });
}

// Usage:
// await indexArticle("kb-042", "Restart the engine after checking battery connection.",
//   { category: "troubleshooting", updatedAt: "2026-06-01" });
// const results = await searchArticles("car won't start", "troubleshooting");

Trade-offs and Pitfalls

The most consequential mistake teams make is treating chunking as an afterthought rather than a decision that directly determines embedding quality. Because an embedding compresses an entire piece of text into a single fixed-length vector, a chunk that's too long forces the model to average together several distinct ideas into one vector, diluting the specific meaning a query might be looking for; a chunk that's too short loses the surrounding context needed to make the fragment meaningful on its own. There is no universal correct chunk size - it depends on the structure of the source material and the kind of queries the system needs to answer - which is precisely why this needs deliberate validation against representative queries rather than a default setting copied from a tutorial.

A second pitfall is assuming semantic similarity is the same thing as relevance for the task at hand. Cosine similarity measures how alike two pieces of text are in the vector space the model was trained to produce, and that space encodes whatever notion of similarity the training data emphasized - which is not always what an application actually needs. A support ticket asking "how do I cancel my subscription" and one asking "how do I downgrade my subscription" may embed as highly similar because they share topic and structure, even though a system routing tickets by intent needs to treat them very differently. Embeddings are a powerful default similarity signal, not a guarantee of task-appropriate relevance, and systems that rely on them exclusively, without any reranking or downstream validation, often surface results that are topically related but practically wrong for the user's actual need.

Best Practices for Working with Embeddings

Choose an embedding model based on evaluation against your actual data and task, not solely on a general leaderboard position. Benchmarks like MTEB are genuinely useful for narrowing a starting shortlist, since they cover a broad range of tasks and reveal which models are strong at retrieval specifically versus clustering or classification, but a model's aggregate benchmark performance does not guarantee it will perform equally well on a specific domain's vocabulary and structure. Building even a small evaluation set - a few dozen representative queries with known correct matches from your own corpus - and testing candidate models against it directly will surface domain-specific gaps that a generic leaderboard cannot.

Combine embeddings with a reranking step for anything where precision matters more than raw recall. A common and effective pattern retrieves a broader set of candidates using fast vector similarity search, then passes that smaller candidate set through a more expensive but more accurate reranking model (a cross-encoder, which directly compares the query and each candidate rather than comparing pre-computed vectors) to reorder them by finer-grained relevance. This two-stage retrieve-then-rerank pattern is standard practice in production search systems precisely because it captures most of the speed of embedding-based retrieval while correcting for the cases where pure vector similarity ranks a topically-related but practically-wrong result too highly.

Keep embedding generation, storage, and retrieval decoupled from any single vector database vendor where reasonably possible, and version your embeddings deliberately. Re-embedding an entire corpus is often necessary when a model is upgraded or replaced, since vectors from different model versions are generally not directly comparable to each other - mixing embeddings generated by two different models in the same index silently degrades retrieval quality in ways that are hard to diagnose after the fact. Treating "which embedding model version indexed this data" as tracked metadata, not an implicit assumption, makes this kind of migration a deliberate, auditable operation rather than a quiet source of production regressions.

Analogies and Mental Models

The clearest way to build intuition for embedding space is to think of it as a well-organized library, but one organized by meaning instead of alphabetically by title. In an alphabetically-organized library, a book about car repair and a book about vehicle maintenance might sit on entirely different shelves, simply because their titles start with different letters - this is the failure mode of keyword search. In a library organized by meaning, those two books sit right next to each other on the shelf, regardless of their exact titles, because a librarian (the embedding model) has already read and understood both and placed them according to what they're actually about. Retrieval becomes "look at what's physically nearby on the shelf" rather than "search for matching words in the title", which is exactly the shift from lexical to semantic search that embeddings enable.

Matryoshka Representation Learning is well captured by the nesting-dolls metaphor its name is drawn from directly: a full embedding vector contains, within its first N dimensions, a smaller but still coherent and meaningful summary of the same information, the way each doll in a matryoshka set contains a smaller but still complete doll inside it. This is why truncating an MRL-trained embedding to a shorter prefix still produces something usable for coarse filtering, rather than producing noise - the smaller representation was explicitly trained to remain meaningful on its own, not simply chopped off after the fact from a model that was never designed with truncation in mind.

The 80/20 of Working With Embeddings

A small number of decisions determine the overwhelming majority of an embedding-based system's real-world quality. Getting chunking right - sized and structured to match the actual source material rather than a default setting - is the single highest-leverage decision, because every downstream capability, from retrieval accuracy to citation quality, is bounded by how well individual chunks capture coherent, self-contained meaning. Choosing an embedding model validated against your own representative queries, rather than chosen purely from a general leaderboard, is the second highest-leverage decision, since domain mismatch between a model's training data and your actual content is one of the most common causes of underwhelming retrieval quality that's difficult to diagnose after the fact.

The third disproportionately valuable practice is adding a reranking step wherever retrieval precision genuinely matters to the user experience, since it corrects for exactly the class of error - topically similar but practically wrong - that pure vector similarity is structurally prone to. Everything beyond these three - exotic distance metrics, elaborate multi-vector retrieval schemes, fine-tuning a custom embedding model from scratch - adds real value for specific, mature use cases, but is refinement layered on top of a foundation that these three decisions already establish. Teams building their first embedding-based system get disproportionately more value from getting chunking, model selection, and reranking right than from optimizing anything further downstream.

Key Takeaways

Treat chunking as a deliberate design decision validated against your actual source material, not a default parameter copied from a tutorial.
Choose an embedding model by testing it against a small, representative evaluation set from your own domain, using a general benchmark like MTEB only to narrow the initial shortlist.
Add a reranking step whenever retrieval precision matters, since embeddings alone conflate topical similarity with task-specific relevance.
Version your embeddings explicitly and re-embed your corpus when switching model versions, since vectors from different models are generally not directly comparable.
Use cosine similarity as your default comparison metric unless a specific model's documentation recommends dot product, and understand which metric your vector database is actually configured to use.

Conclusion

Embeddings are the piece of infrastructure that made semantic search, RAG, and a wide range of similarity-based features practical at scale, by compressing the expensive work of understanding meaning into a fixed-length vector that can be compared with fast, simple arithmetic. Understanding what that vector actually represents - a position in a space shaped by contrastive training on specific data, compared using cosine similarity or a related metric, retrieved efficiently through an approximate nearest-neighbor index - turns embeddings from an opaque API call into a tool an engineer can actually reason about and debug.

The systems that use embeddings well are rarely the ones using the fanciest model; they're the ones that got the fundamentals right: chunking matched to the actual content, a model validated against real queries from the actual domain, and a reranking step to correct for the gap between semantic similarity and task relevance. Everything else - dimensionality tricks, exotic indexing structures, multimodal extensions - is genuine value added on top of that foundation, not a substitute for getting it right in the first place.

References

Massive Text Embedding Benchmark (MTEB) - Muennighoff et al., 2023: https://arxiv.org/abs/2210.07316
Matryoshka Representation Learning - Kusupati et al., 2022: https://arxiv.org/abs/2205.13147
OpenAI - Embeddings API documentation: https://platform.openai.com/docs/guides/embeddings
Cohere - Embed API documentation: https://docs.cohere.com/docs/embeddings
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks - Reimers & Gurevych, 2019: https://arxiv.org/abs/1908.10084
HNSW: Efficient and Robust Approximate Nearest Neighbor Search - Malkov & Yashunin, 2018: https://arxiv.org/abs/1603.09320
Qdrant - official documentation: https://qdrant.tech/documentation/
pgvector - PostgreSQL vector extension: https://github.com/pgvector/pgvector