Introduction
Two capabilities quietly do most of the load-bearing work in a mature LLM application, and neither of them is the model call itself. The first is embeddings - turning content into vectors so a system can find what's relevant by meaning rather than exact wording. The second is evaluation - turning "does this output seem okay" into a measurable, trackable signal. Prompting sits between them: it's the lever an engineer pulls to shape what a model produces, and it only becomes a genuine engineering practice, rather than trial and error, once there's an evaluation pipeline capable of telling you whether a given prompt change actually helped.
These three concepts are usually taught separately, but in practice they form a single feedback loop. Embeddings retrieve the context a prompt is built from; a prompt shapes what the model produces; evaluation scores that output, often itself relying on embedding-based similarity as one of its metrics; and the results of evaluation are what tell you whether to change the prompt, the retrieval strategy, or the embedding model in the first place. This article treats embeddings and evaluation as the two connected foundations of AI engineering they actually are, working through what each does technically, where they intersect directly, and the concrete engineering practices that make the whole loop trustworthy rather than just plausible-looking.
Context: Why Representation and Measurement Are Both Non-Negotiable
An LLM's raw capability - its ability to generate fluent, plausible text - is necessary but nowhere near sufficient for a production system, because fluency and correctness are entirely different properties. A model can write a confident, well-structured, grammatically perfect answer that is completely wrong, either because it lacked the specific information needed to answer correctly or because nobody checked whether the answer was actually right before it shipped. Embeddings address the first failure mode, by giving a system a way to retrieve the specific, current, or private information a model's training data doesn't contain. Evaluation addresses the second, by giving a team a way to systematically check whether outputs are actually correct, rather than relying on how confident and fluent they sound.
The two failure modes these techniques address are easy to conflate but genuinely distinct, and it's worth being precise about the difference. A system with excellent retrieval but no evaluation can ground its answers in the right facts and still occasionally misinterpret them, contradict them, or answer a different question than the one asked - problems evaluation exists to catch. A system with rigorous evaluation but poor retrieval will accurately and consistently measure that its answers are wrong, without that measurement, on its own, fixing the underlying cause - which is why evaluation results need to be diagnostic enough to point back at whether the failure originated in retrieval, generation, or somewhere else. Neither capability substitutes for the other; a mature AI engineering practice needs both operating together.
The third reason to treat these as unified foundations rather than separate specialties is that they reuse the same underlying mathematical tool. Embedding-based semantic similarity is not just a retrieval mechanism - it's also one of the most common building blocks used inside evaluation pipelines themselves, to check whether a generated answer is semantically close to a reference answer, to detect near-duplicate test cases when curating a golden dataset, or to cluster failure cases by underlying cause during error analysis. Understanding embeddings deeply pays off twice: once for building retrieval, and again for building the evaluation infrastructure that checks whether that retrieval, and everything built on top of it, is actually working.
Deep Technical Explanation: How Embeddings and Evaluation Actually Work
An embedding model maps a piece of text into a fixed-length vector positioned in a high-dimensional space such that semantically similar inputs land close together, typically measured using cosine similarity - the angle between two vectors, producing a score largely insensitive to text length. Embedding models are trained with a contrastive objective, shown pairs of similar and dissimilar examples and adjusted so the model's output vectors reflect that similarity structure, which is why the specific data a model was trained on matters: a model trained on general web text represents everyday semantic similarity well but can underperform on specialized domains like legal or medical text, where the industry-standard Massive Text Embedding Benchmark (MTEB) is the common reference point for comparing candidate models across retrieval, classification, and clustering tasks.
Evaluation, by contrast, is fundamentally about scoring output quality against some standard, and the standards fall into a few distinct families. Reference-based metrics compare an output directly against a known-correct answer - exact match, or embedding-based semantic similarity between the generated and reference text - and work well for tasks with a genuinely correct answer. LLM-as-judge techniques, formalized in frameworks like G-Eval, use a separate model call to score an output against a natural-language rubric, which fits open-ended generation tasks where multiple differently-worded outputs can all be correct and no single fixed reference captures that. For retrieval-augmented generation specifically, metrics like faithfulness (is the answer actually supported by retrieved context), context precision and recall (is retrieval itself working), and answer relevancy (does the response address the actual question) let a team isolate whether a failure originates in retrieval or in generation - a diagnostic capability that depends directly on the embedding-based retrieval layer being instrumented well enough to inspect separately from the final answer.
Where Embeddings and Evaluation Intersect
The most direct intersection between these two disciplines is using embedding-based semantic similarity as an evaluation metric in its own right. Rather than requiring an exact string match between a generated answer and a reference answer - which fails for any output that's correct but differently worded - a semantic similarity score embeds both the generated and reference text and compares them by cosine similarity, catching correctness across paraphrase in a way exact-match scoring cannot. This is a fast, cheap, fully deterministic metric relative to an LLM-as-judge call, which makes it attractive for high-volume regression testing, though it comes with the same caveat that applies to embeddings generally: semantic similarity is not the same thing as task-appropriate correctness, and a reference-similarity score can be fooled by an answer that's topically similar but factually wrong in a specific detail.
A second intersection point is using embeddings to curate and maintain a golden evaluation dataset. As a team accumulates test cases drawn from real production failures, the dataset can quietly become redundant - many near-duplicate cases testing the same underlying failure mode, while entire categories of real user queries go untested. Embedding every test case and clustering them by similarity surfaces this redundancy directly: dense clusters indicate over-tested scenarios, while embedding the full space of real production queries and comparing it against the embedded test set reveals gaps - areas of real usage with no corresponding test coverage at all. This turns dataset curation from a subjective, manual review process into something that can be checked quantitatively.
A third intersection is more subtle but increasingly relevant as evaluation pipelines mature: error analysis over evaluation results themselves. When an evaluation run produces a batch of low-scoring outputs, embedding the associated inputs (or the judge's rationale text) and clustering them can reveal that failures aren't randomly distributed - they cluster around a specific category of question, a specific retrieval pattern, or a specific prompt structure. This turns "12% of outputs failed evaluation this week" into a specific, actionable finding - "failures cluster around questions referencing recently updated policy documents" - which is a far more useful signal for deciding what to actually fix than an aggregate pass rate on its own.
Implementation: Combining Embedding-Based and LLM-Judge Scoring
A production evaluation pipeline rarely relies on a single scoring method; it typically combines a cheap, deterministic embedding-similarity check with a more expensive but more discerning LLM-as-judge pass, running the cheaper check first to filter or flag candidates before committing judge-model tokens to a smaller, more targeted set. The Python example below shows this two-stage pattern applied to a batch of evaluation cases.
# hybrid_eval.py
import numpy as np
from dataclasses import dataclass
from openai import OpenAI
client = OpenAI()
@dataclass
class EvalCase:
question: str
reference_answer: str
generated_answer: str
def embed(texts: list[str]) -> np.ndarray:
response = client.embeddings.create(input=texts, model="text-embedding-3-small")
return np.array([item.embedding for item in response.data])
def cosine_sim(a: np.ndarray, b: np.ndarray) -> float:
return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
def semantic_similarity_score(case: EvalCase) -> float:
vectors = embed([case.reference_answer, case.generated_answer])
return cosine_sim(vectors[0], vectors[1])
def llm_judge_score(case: EvalCase) -> dict:
prompt = f"""Rate how well the response answers the question, from 1 to 5,
compared to the reference answer's meaning (not exact wording).
Question: {case.question}
Reference answer: {case.reference_answer}
Generated answer: {case.generated_answer}
Return JSON only: {{"score": <1-5>, "reasoning": "<one sentence>"}}"""
result = client.chat.completions.create(
model="gpt-5-mini",
messages=[{"role": "user", "content": prompt}],
temperature=0,
)
import json
return json.loads(result.choices[0].message.content)
def run_hybrid_eval(cases: list[EvalCase], similarity_threshold: float = 0.85) -> list[dict]:
results = []
for case in cases:
sim_score = semantic_similarity_score(case)
# Only spend judge-model tokens on cases the cheap metric flags as
# uncertain or likely problematic - high-similarity cases are
# assumed correct without a judge call, saving cost at scale.
if sim_score >= similarity_threshold:
results.append({"case": case, "similarity": sim_score, "judged": False})
continue
judge_result = llm_judge_score(case)
results.append({
"case": case,
"similarity": sim_score,
"judged": True,
"judge_score": judge_result["score"],
"judge_reasoning": judge_result["reasoning"],
})
return results
Curating a golden dataset by finding coverage gaps is the other pattern worth showing directly, since it turns dataset maintenance from manual review into a repeatable, scriptable check. The TypeScript example below embeds a production query log and an existing golden dataset, then surfaces production queries that have no close match in the current test coverage.
// coverageGapAnalysis.ts
import OpenAI from "openai";
const openai = new OpenAI();
async function embedBatch(texts: string[]): Promise<number[][]> {
const response = await openai.embeddings.create({
input: texts,
model: "text-embedding-3-small",
});
return response.data.map((d) => d.embedding);
}
function cosineSimilarity(a: number[], b: number[]): number {
const dot = a.reduce((sum, v, i) => sum + v * b[i], 0);
const normA = Math.sqrt(a.reduce((s, v) => s + v * v, 0));
const normB = Math.sqrt(b.reduce((s, v) => s + v * v, 0));
return dot / (normA * normB);
}
async function findCoverageGaps(
productionQueries: string[],
goldenDatasetQuestions: string[],
gapThreshold = 0.75
): Promise<string[]> {
const [prodVectors, goldenVectors] = await Promise.all([
embedBatch(productionQueries),
embedBatch(goldenDatasetQuestions),
]);
const gaps: string[] = [];
prodVectors.forEach((prodVec, i) => {
const maxSimilarity = Math.max(
...goldenVectors.map((goldenVec) => cosineSimilarity(prodVec, goldenVec))
);
// A production query with no close match in the golden dataset
// represents a real-world case the eval suite doesn't currently cover.
if (maxSimilarity < gapThreshold) {
gaps.push(productionQueries[i]);
}
});
return gaps;
}
Trade-offs and Pitfalls
The most common mistake is treating embedding-based similarity as a complete substitute for LLM-as-judge scoring, when the two catch genuinely different failure classes. A semantically similar answer can still be factually wrong in a specific, important detail - a date, a number, a name - that cosine similarity, which reflects overall topical closeness rather than granular factual precision, simply won't detect. Relying solely on similarity-based scoring for tasks where precise factual correctness matters produces a false sense of coverage: the metric looks healthy while missing exactly the errors most likely to matter to an actual user.
A second pitfall specific to the hybrid pattern shown above is miscalibrating the threshold that decides when to escalate from the cheap similarity check to the more expensive judge call. Setting the threshold too high sends nearly every case to the judge, eliminating the cost savings the two-stage design was meant to provide; setting it too low lets genuinely incorrect but superficially similar answers slip through without ever being judged. This threshold needs to be validated against a labeled sample - cases with known-correct judge verdicts - rather than picked arbitrarily, and it should be revisited whenever the underlying embedding model or the nature of the application's typical answers changes meaningfully.
A third, more structural pitfall is failing to version and track which embedding model produced which set of vectors, both in the retrieval system and in the evaluation pipeline. Vectors from two different embedding models are generally not comparable to each other, and if an evaluation pipeline's semantic-similarity metric silently starts using a different or updated embedding model than the one used when a threshold was originally calibrated, historical evaluation scores and current ones are no longer measuring the same thing - a source of confusing, hard-to-diagnose evaluation drift that has nothing to do with the actual application's quality and everything to do with an unversioned dependency underneath the measurement layer itself.
Best Practices for Building an Integrated Pipeline
Instrument retrieval and generation as separately measurable stages from the start, so an evaluation failure can be traced to whichever stage actually caused it rather than requiring a fresh investigation every time. This means capturing, alongside every generated output, exactly which chunks were retrieved and what their similarity scores were - the same data needed to compute retrieval-specific metrics like context precision and recall - so that a low-scoring output can be immediately checked against whether retrieval surfaced the right material in the first place, before assuming the generation step is at fault.
Build the cheap-metric-first, judge-second pattern deliberately into any evaluation pipeline running at meaningful scale, and validate the escalation threshold against labeled ground truth rather than picking it by intuition. This keeps evaluation both affordable and accurate: the majority of clearly-fine or clearly-broken cases get resolved by the fast, deterministic similarity check, while judge-model budget is spent specifically on the ambiguous middle where a more discerning, rubric-based assessment actually adds value. Pair this with periodic recalibration - re-checking the threshold and the judge's own scoring behavior against a fresh sample of human-reviewed cases - since both the application's typical outputs and any underlying model dependencies can drift over time in ways a fixed, never-revisited threshold won't catch.
Analogies and Mental Models
A useful way to think about the relationship between embeddings and evaluation is the distinction between a metal detector and a forensic lab. A metal detector (embedding-based similarity) is fast, cheap, and excellent at a coarse first pass - quickly telling you which spots are worth digging into further, out of a huge search area. It cannot, on its own, tell you exactly what's buried or whether it matters. A forensic lab (LLM-as-judge evaluation) is slower and more expensive per sample, but gives a detailed, reasoned assessment of exactly what was found. A competent excavation uses the detector to narrow down where to dig, then sends only the promising finds to the lab - exactly the two-stage pattern a hybrid evaluation pipeline implements, and using the expensive tool on every square inch of ground would be needlessly costly, while relying on the detector alone would miss everything it can't detect.
The feedback loop connecting embeddings, prompting, and evaluation is well captured by a thermostat system rather than a single sensor. A thermostat doesn't just measure temperature once and stop - it continuously measures, compares against a target, and adjusts the heating or cooling system accordingly, in an ongoing loop. Embeddings are the sensor (retrieving what's relevant), prompting is the mechanism that acts on what's sensed (shaping the model's response), and evaluation is the comparison against the target that decides whether to keep the current settings or adjust them. A system that only measures once, before launch, is a thermostat with a broken feedback loop - it might have been calibrated correctly on day one, but it has no way of noticing or correcting for drift afterward.
The 80/20 of AI Engineering's Measurement Layer
A small number of practices account for most of the reliability an integrated embeddings-and-evaluation setup can deliver. Instrumenting retrieval and generation as separately measurable stages is the single highest-leverage practice, because it's what makes every evaluation result diagnostic rather than merely descriptive - the difference between knowing something is wrong and knowing specifically what to fix. Building a two-stage evaluation pipeline - cheap embedding-based similarity first, LLM-as-judge scoring reserved for ambiguous or flagged cases - is the second highest-leverage practice, since it's what makes running evaluation continuously, at real production scale, financially sustainable rather than a periodic, expensive exercise reserved for pre-launch checks.
The third disproportionately valuable practice is using embeddings to actively maintain the evaluation dataset itself - surfacing redundant test cases and real coverage gaps - rather than letting a golden dataset grow passively and unevenly over time. Everything beyond these three - exotic multi-judge ensembles, fine-tuned custom embedding models, elaborate per-dimension scoring taxonomies - adds genuine value for mature, high-stakes systems, but is refinement layered on top of a foundation these three practices already establish. Teams that get the diagnostic instrumentation, the cost-aware evaluation cadence, and the deliberate dataset maintenance right will see disproportionately more return than teams chasing more sophisticated techniques in either discipline before getting these fundamentals in place.
Key Takeaways
- Instrument retrieval and generation as separately measurable stages so evaluation failures can be traced to their actual cause, not just flagged as "something went wrong."
- Use embedding-based semantic similarity as a fast, cheap first-pass evaluation metric, and reserve LLM-as-judge scoring for the ambiguous cases it flags, rather than running an expensive judge on every single output.
- Validate any similarity threshold used to escalate between the two stages against a labeled, human-reviewed sample - don't pick it by intuition.
- Use embeddings to actively maintain your golden evaluation dataset: cluster it to find redundancy, and compare it against real production queries to find coverage gaps.
- Version and track which embedding model produced which vectors in both your retrieval system and your evaluation pipeline, since silently mixing model versions invalidates historical comparisons.
Conclusion
Embeddings and evaluation are frequently taught as separate specialties, but in a working AI engineering practice they form a single loop: embeddings retrieve what a prompt needs to ground its answer, the prompt shapes what the model actually produces, and evaluation - often itself built partly on embedding-based similarity - measures whether that output was actually good, feeding back into decisions about what to change next. Treating them as connected disciplines rather than isolated skills is what turns a collection of individually reasonable techniques into a system that actually improves over time rather than one that was merely tuned once and left alone.
The practical payoff of this integrated view is concrete: retrieval failures get separated from generation failures instead of being lumped into one vague "quality problem," evaluation stays affordable enough to run continuously rather than only before a launch, and a team's evaluation dataset actively tracks real usage instead of quietly drifting away from it. None of this requires exotic technique - it requires treating representation and measurement as the two foundations they actually are, and building the connective tissue between them deliberately rather than leaving it implicit.
References
- Massive Text Embedding Benchmark (MTEB) - Muennighoff et al., 2023: https://arxiv.org/abs/2210.07316
- G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment - Liu et al., 2023: https://arxiv.org/abs/2303.16634
- RAGAS: Automated Evaluation of Retrieval Augmented Generation - Es et al., 2023: https://arxiv.org/abs/2309.15217
- OpenAI - Embeddings API documentation: https://platform.openai.com/docs/guides/embeddings
- DeepEval - open-source LLM evaluation framework documentation: https://deepeval.com/docs/getting-started
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks - Reimers & Gurevych, 2019: https://arxiv.org/abs/1908.10084
- Matryoshka Representation Learning - Kusupati et al., 2022: https://arxiv.org/abs/2205.13147
- HELM: Holistic Evaluation of Language Models - Stanford CRFM: https://crfm.stanford.edu/helm/