Introduction
"Which model should we use for this feature" is a question most teams answer badly: someone tries a handful of prompts against two or three models in a chat playground, forms an impression, and ships whichever one felt better. This works about as well as it sounds - a handful of manually-tried examples is a small, unrepresentative, and typically non-blind sample, and "felt better" is not a measurement. Model selection is a genuine engineering decision with real cost, latency, and quality trade-offs, and it deserves the same experimental rigor a team would apply to any other consequential technical choice: a clearly defined hypothesis, a representative test set, hard metrics, and enough statistical care to know whether an observed difference is real or just noise.
OpenRouter is a useful piece of infrastructure for this specific problem because it exposes dozens of models from different providers through a single, OpenAI-compatible endpoint, which means switching which model an experiment targets is a configuration change rather than a new SDK integration. This article works through how to design a genuine model A/B test - not a vibes-based comparison - using OpenRouter as the unified access layer, covering experiment design, the hard metrics worth tracking, how to determine whether a measured difference is statistically meaningful, and the practical pitfalls that undermine model comparisons even when the intent behind them is good.
Context: Why Model Selection Needs an Experimental Discipline
The temptation to skip rigorous experimentation is understandable, since running a genuine A/B test takes real setup effort compared to trying a few prompts by hand. But the cost of skipping it is a decision made on a sample size of a handful of hand-picked, non-representative examples, run once, by one person, with no measure of whether the observed difference would hold up against the actual diversity of real production traffic. This is precisely the same failure mode that motivated the shift toward rigorous evaluation pipelines more broadly - a model that looks better on five chosen examples can easily be worse on the actual distribution of real user input, and without a systematic comparison, a team has no way to know which is actually true.
The second reason this deserves real experimental discipline is that model comparisons involve multiple, often competing, dimensions that don't move together. A more capable model might produce meaningfully better output on complex reasoning tasks while costing several times more per request and adding real latency; a cheaper, faster model might be entirely adequate for a simpler subset of the same feature's traffic. Without measuring cost, latency, and quality as distinct, independently tracked dimensions, a comparison collapses into a vague single judgment - "model A seemed better" - that can't actually inform a decision like "route 80% of traffic to the cheaper model and escalate the remaining 20% to the more capable one," which is often the genuinely correct architecture once the trade-offs are actually measured.
The third reason is statistical: LLM outputs are non-deterministic, and any single comparison between two models on a small number of examples is vulnerable to noise that has nothing to do with which model is actually better. A model that wins on 6 out of 10 hand-picked examples might simply be within the range of normal variation for two models of genuinely similar quality - without a large enough sample and a basic statistical framework for interpreting the result, a team can easily conclude a difference exists when it doesn't, or miss a real difference because a small sample happened to land close to a tie by chance. This is precisely the discipline that makes an "A/B test" different from an anecdotal comparison, and it's exactly what's missing from the chat-playground approach to model selection.
Deep Technical Explanation: Designing a Rigorous Model Experiment
A sound model experiment starts with a fixed, representative test set - the same golden dataset discipline used for evaluation more broadly - rather than an ad hoc set of prompts invented for the comparison itself. Every model under test needs to be run against the exact same set of inputs, since comparing different models against different examples introduces a confound that makes the comparison meaningless regardless of how the results come out. This test set should be drawn from real production traffic patterns wherever possible, since a model that performs well on a curated set of "clean" examples may behave very differently against the actual messiness of real user input - ambiguous phrasing, unusual formatting, edge cases the curated set didn't anticipate.
OpenRouter's architectural fit for this kind of experiment comes from its single-endpoint, model-string-based design. OpenRouter exposes one OpenAI-compatible endpoint where any participating provider's model is selectable by a single model string, which means existing OpenAI SDK-based code becomes largely drop-in compatible by swapping the base URL and the model identifier. For an experiment runner, this means the only thing that changes between testing GPT-5, Claude Sonnet 5, and an open-weight model is the model string passed in the request - the request and response handling code stays identical, which meaningfully reduces the engineering overhead of running the same experiment across many providers compared to integrating each provider's native SDK separately. The response itself, along with OpenRouter's model routing behavior of falling back to alternate providers on a server error or rate limit, means the actual model and provider that served a given request can and should be logged from the response rather than assumed from the request parameters alone, which matters for keeping an experiment's results attributable to the model that actually generated them.
The three hard metrics worth tracking systematically are cost, latency, and quality, each requiring its own measurement approach. Cost and token usage are directly available in the response payload for each request and should be logged per-call rather than estimated after the fact. Latency should be measured both as total round-trip time and, for streaming responses, as time-to-first-token separately, since these represent different user-facing experiences and can behave very differently across providers and models. Quality is the hardest of the three to measure objectively and typically requires the same scoring methods used in evaluation pipelines generally - reference-based metrics for tasks with a clear correct answer, or LLM-as-judge scoring against a defined rubric for open-ended generation - applied identically across every model being compared so the resulting quality scores are actually comparable to each other.
Implementation: Building a Multi-Model Experiment Runner
The Python example below implements an experiment runner that sends the same fixed test set to multiple models via OpenRouter, capturing cost, latency, and the raw output needed for downstream quality scoring, all in a structured format suitable for statistical analysis afterward.
# model_experiment.py
import time
import json
from dataclasses import dataclass, asdict
from openai import OpenAI
client = OpenAI(
base_url="https://openrouter.ai/api/v1",
api_key="YOUR_OPENROUTER_API_KEY",
)
@dataclass
class ExperimentResult:
model: str
test_case_id: str
prompt: str
output: str
latency_ms: float
prompt_tokens: int
completion_tokens: int
cost_usd: float | None
def run_single_call(model: str, test_case_id: str, prompt: str) -> ExperimentResult:
start = time.time()
response = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
extra_body={"usage": {"include": True}}, # request cost data in the response
)
latency_ms = (time.time() - start) * 1000
usage = response.usage
cost = getattr(usage, "cost", None) # OpenRouter includes cost when requested
return ExperimentResult(
model=response.model, # the actual model/provider that served the request
test_case_id=test_case_id,
prompt=prompt,
output=response.choices[0].message.content,
latency_ms=latency_ms,
prompt_tokens=usage.prompt_tokens,
completion_tokens=usage.completion_tokens,
cost_usd=cost,
)
def run_experiment(models: list[str], test_cases: list[dict]) -> list[ExperimentResult]:
results = []
for model in models:
for case in test_cases:
result = run_single_call(model, case["id"], case["prompt"])
results.append(result)
return results
def save_results(results: list[ExperimentResult], path: str):
with open(path, "w") as f:
json.dump([asdict(r) for r in results], f, indent=2)
# Usage:
# test_cases = [{"id": "q1", "prompt": "Summarize the attached refund policy..."}, ...]
# results = run_experiment(
# models=["openai/gpt-5-mini", "anthropic/claude-sonnet-5", "google/gemini-2.5-flash"],
# test_cases=test_cases,
# )
# save_results(results, "experiment_results.json")
Once results are collected, the analysis needs to answer a specific statistical question: is an observed difference between two models' quality scores likely to be real, or could it plausibly be explained by chance given the sample size? The TypeScript example below implements a basic two-sample comparison using a t-test approximation, appropriate for comparing mean quality scores (for instance, from an LLM-as-judge rubric scored 1-5) between two models on the same test set.
// statisticalComparison.ts
interface ScoredResult {
model: string;
testCaseId: string;
qualityScore: number; // e.g. 1-5 from an LLM-as-judge rubric
}
function mean(values: number[]): number {
return values.reduce((sum, v) => sum + v, 0) / values.length;
}
function stdDev(values: number[], meanValue: number): number {
const variance =
values.reduce((sum, v) => sum + (v - meanValue) ** 2, 0) / (values.length - 1);
return Math.sqrt(variance);
}
interface ComparisonResult {
modelA: string;
modelB: string;
meanA: number;
meanB: number;
tStatistic: number;
meaningfulDifference: boolean; // simple heuristic threshold, see note below
}
function compareModels(
resultsA: ScoredResult[],
resultsB: ScoredResult[]
): ComparisonResult {
const scoresA = resultsA.map((r) => r.qualityScore);
const scoresB = resultsB.map((r) => r.qualityScore);
const meanA = mean(scoresA);
const meanB = mean(scoresB);
const sdA = stdDev(scoresA, meanA);
const sdB = stdDev(scoresB, meanB);
// Welch's t-statistic for two independent samples with unequal variance -
// appropriate here since different models may have very different score
// variability, not just different means.
const standardError = Math.sqrt(
(sdA ** 2) / scoresA.length + (sdB ** 2) / scoresB.length
);
const tStatistic = (meanA - meanB) / standardError;
// A |t| above roughly 2 corresponds to a conventional significance
// threshold for reasonably sized samples - treat this as a starting
// heuristic, not a substitute for computing an actual p-value with a
// statistics library when the decision genuinely matters.
return {
modelA: resultsA[0].model,
modelB: resultsB[0].model,
meanA,
meanB,
tStatistic,
meaningfulDifference: Math.abs(tStatistic) > 2,
};
}
Trade-offs and Pitfalls
The most common mistake is comparing models on too small a sample and treating the result as conclusive. A handful of test cases, even run systematically rather than by hand, doesn't provide enough statistical power to distinguish a genuine quality difference from ordinary variance - the t-test approach shown above becomes meaningfully more trustworthy as sample size grows, and a team drawing a firm conclusion from ten or twenty test cases is at real risk of a false positive or false negative. Determining an appropriate sample size in advance, based on the size of a difference that would actually matter for the decision, is worth doing deliberately rather than running whatever number of test cases happened to be convenient.
A second pitfall is failing to control for confounding variables between models being compared - prompt formatting that happens to suit one model's training better than another's, or a temperature setting left at a different default across providers. Every variable other than the model itself needs to be held constant across the comparison, including the exact prompt template, sampling parameters, and any system instructions, since a difference attributed to "model A is better" can easily actually be "model A happens to handle this specific prompt format better," a genuinely different and less useful finding.
A third pitfall specific to routing through an aggregator like OpenRouter is not verifying which underlying provider and model version actually served each request. Because OpenRouter can fall back to alternate providers on an error or rate limit, and because a model string can sometimes map to a model that receives updates over time, an experiment that doesn't log the actual model field from each response risks silently mixing results from different underlying providers or model versions under what the analysis assumes is a single, consistent condition - undermining the comparison's validity without producing any visible error.
Best Practices for Running Model Experiments
Define hard success metrics and a decision threshold before running the experiment, not after seeing the results. Deciding in advance what magnitude of cost difference, latency difference, or quality difference would actually change the decision - rather than examining the results first and then deciding whether they seem meaningful - avoids the natural human tendency to rationalize whatever result appears as significant after the fact. This is the same discipline that underlies pre-registration in more formal experimental research, and it applies just as directly to a model comparison run for an engineering decision.
Hold every variable other than the model itself constant across the comparison - identical prompts, identical sampling parameters, identical test set - and log the actual model and provider that served every single request rather than assuming it matches the request parameters. Combine this with running each test case multiple times per model where feasible, since LLM output variance means a single response to a single prompt is one sample from a distribution, not a definitive characterization of that model's typical behavior on that input.
Report results with the actual uncertainty attached, not just a single point estimate difference between models. A finding like "model A scored higher on average, and this difference is unlikely to be due to chance given the sample size" is meaningfully more trustworthy and more actionable than "model A scored 4.2 versus model B's 3.9," which omits any indication of whether that gap reflects a real difference or ordinary variance. Decision-makers acting on a model comparison deserve to know how confident the underlying data actually supports the recommendation being made.
Analogies and Mental Models
Running a rigorous model comparison is well captured by the difference between a controlled clinical trial and a doctor's personal impression that a treatment "seems to work" based on a few patients. A doctor's anecdotal impression, formed from a handful of cases seen in their own practice, isn't worthless, but it's also not a substitute for a properly controlled trial with a large enough sample, a consistent protocol, and statistical analysis of whether an observed effect could plausibly be due to chance. Trying five prompts against two models in a chat interface is the doctor's anecdotal impression; a systematic experiment with a fixed test set, controlled variables, and statistical comparison of results is the clinical trial - and the stakes of a production model choice deserve the latter, not the former.
OpenRouter's role in this process is well captured by a universal testing rig used to benchmark different engines under otherwise identical conditions. A mechanic benchmarking several car engines wants to control every variable except the engine itself - same chassis, same fuel, same test track, same driver - so that any measured difference in performance can be attributed to the engines and nothing else. OpenRouter's single, consistent request and response format across many different model providers plays the same role for an LLM experiment: it's the consistent testing rig that lets a team swap only the "engine" (the model) between runs, rather than also changing the surrounding harness every time a different provider's native SDK is substituted in.
The 80/20 of Systematic Model Experimentation
A small number of practices account for most of the rigor a model comparison actually needs. Using a fixed, representative test set drawn from real usage patterns, run identically against every model under comparison, is the single highest-leverage practice, because it's the foundation that makes any subsequent metric - cost, latency, quality score - a genuinely comparable number rather than an artifact of different test conditions. Logging cost, latency, and the actual serving model from every individual request, rather than relying on aggregate estimates, is the second highest-leverage practice, since it's what makes granular analysis - which specific test cases drove a cost difference, whether a particular provider's fallback routing affected results - possible after the fact rather than only available as a single opaque summary number.
The third disproportionately valuable practice is applying basic statistical reasoning to the resulting comparison - checking whether an observed difference in scores is large relative to the sample's natural variance - rather than treating any measured difference, however small or however noisy, as a definitive finding. Everything beyond these three - sophisticated multi-armed bandit allocation across models in production, elaborate per-segment analysis by user cohort, continuous online experimentation infrastructure - adds genuine value for mature, high-traffic systems, but is refinement layered on top of a foundation these three practices already establish. Teams running their first systematic model comparison get disproportionately more value from getting the test set, the logging, and the statistical framing right than from any more elaborate experimentation infrastructure.
Key Takeaways
- Use a fixed, representative test set drawn from real usage, and run it identically against every model under comparison - never compare different models against different examples.
- Track cost, latency, and quality as three separate, independently logged metrics per request, not a single vague impression of "which model seemed better."
- Log the actual model and provider that served each request from the response itself, especially when routing through an aggregator like OpenRouter that can fall back between providers.
- Hold every variable other than the model constant - prompt template, sampling parameters, system instructions - so an observed difference can actually be attributed to the model itself.
- Apply basic statistical reasoning (checking whether a score difference is large relative to sample variance) before treating any comparison result as a definitive finding.
Conclusion
Choosing a model for a production feature is a real engineering decision with measurable trade-offs across cost, latency, and quality, and it deserves the same experimental discipline as any other consequential technical choice - not a handful of prompts tried by hand in a chat interface. OpenRouter's single, OpenAI-compatible endpoint across many providers removes a meaningful amount of the integration overhead that would otherwise make testing several models tedious, letting an experiment runner treat the model itself as the one variable under test while everything else - request format, response handling - stays consistent.
The actual rigor, though, comes from the experimental design and the analysis, not from the routing infrastructure: a representative fixed test set, hard metrics logged per request, and basic statistical reasoning about whether an observed difference is real or noise. Teams that build this discipline into their model selection process end up making decisions grounded in evidence proportional to how consequential the decision actually is - which, for a choice that affects cost, latency, and output quality across every request a feature serves, is exactly the level of rigor the decision deserves.
References
- OpenRouter - official documentation: https://openrouter.ai/docs
- OpenRouter - API Reference: https://openrouter.ai/docs/api-reference/overview
- G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment - Liu et al., 2023: https://arxiv.org/abs/2303.16634
- Student's t-test - statistical methodology reference (NIST/SEMATECH e-Handbook of Statistical Methods): https://www.itl.nist.gov/div898/handbook/eda/section3/eda353.htm
- DeepEval - open-source LLM evaluation framework documentation: https://deepeval.com/docs/getting-started
- OpenAI - Chat Completions API reference: https://platform.openai.com/docs/api-reference/chat