Production LLM Architecture: Gateway, Orchestration, RAG, and Observability

A reference architecture for turning a single LLM API call into a production system, using an AI gateway, an orchestration layer, retrieval, structured outputs, and observability as five composable layers around every model request.

Overview

A single direct API call to one model provider breaks down in production because it can't solve routing (provider outages, cost), observability (why did it say that), or evaluation (is the new prompt actually better) problems on its own.
The architecture separates into three planes: a data plane (embeddings, vector databases, retrieval), a control plane (orchestration, prompt templates, schemas, the gateway), and an observability plane (tracing, scoring, datasets, A/B results) - most architectural mistakes come from conflating these planes.
A request flows: app layer -> vector database retrieval -> orchestration assembles a structured prompt -> AI gateway routes to a model provider -> schema validates the response, with every step emitting a trace.
The gateway makes model choice a config parameter, the orchestration layer makes multi-step pipelines explicit and testable, and tracing turns "vibes-based" prompt/model decisions into evidence-based ones.
Fine-tuning sits at the end of this stack, not the start - it's only worth doing once prompting, retrieval, and structured-output enforcement are exhausted and you have an eval dataset to measure it against.

Key Concepts

AI Gateway (Routing Layer)

Sits between the app and model providers behind a single, typically OpenAI-compatible, API contract - model selection becomes a config string change instead of a code change.
Enables three things directly: provider redundancy (fallback to another model on rate-limit/outage), cost-aware routing (cheap models for simple tasks, frontier models for hard ones), and experimentation velocity (swap model string to A/B test).
Example pattern: a tiered fallback wrapper (fast / balanced / frontier) that tries an ordered list of models per tier via OpenRouter and throws only after all candidates in a tier fail.
Caveat baked into the design: it adds a network hop and a dependency on the gateway provider's own uptime and pricing.

Orchestration (Control Plane)

LangChain provides composable primitives (chains, agents, tool-calling, memory) for expressing multi-step control flow in code rather than ad hoc string concatenation.
LlamaIndex specializes in the ingestion/retrieval side - connecting sources, chunking, indexing, querying - and many systems use both together (LlamaIndex for retrieval, LangChain for surrounding logic).
Value is turning an implicit, monolithic prompt into discrete, named steps (retrieve -> rerank -> format -> generate -> parse) that can each be tested, cached, and traced independently - this is what makes downstream traces legible (a span named "retrieval" vs. "generation" localizes latency/cost immediately).
Pitfall: teams over-adopt orchestration frameworks for simple single-call use cases where a direct API call would be more transparent; the abstraction is worth its overhead only once the pipeline has genuinely multiple coordinated steps.

Structured Outputs & Schemas

Solves making model output machine-consumable without brittle string parsing, via JSON mode, function/tool calling, or JSON schema constraints.
Best practice: define the shape once (e.g. with Zod in TypeScript), derive both the schema sent to the model and the runtime validator from the same source, so request and validation can't drift apart.
Critical caveat: provider-side schema enforcement is not infallible - truncated responses, model regressions, or gateway-level inconsistencies across providers can still produce malformed JSON, so re-validating on the receiving end is mandatory, not optional defensive coding.

export const TicketTriage = z.object({
  category: z.enum(["billing", "technical", "account", "other"]),
  priority: z.enum(["low", "medium", "high", "urgent"]),
  requires_human_escalation: z.boolean(),
});
// schema drives both the model-facing JSON schema and the local re-validation
return TicketTriage.parse(JSON.parse(raw));

Retrieval-Augmented Generation (RAG) & Vector Databases

RAG (formalized in Lewis et al., 2020, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks") addresses a model's frozen training-time knowledge by embedding documents, storing vectors, and retrieving nearest matches by similarity (cosine/ANN) at query time.
Vector store choice is an operational trade-off: Postgres + pgvector avoids adding a new dependency at moderate scale; purpose-built stores (Pinecone, Weaviate, Qdrant, Milvus) offer better scale, hybrid (dense + sparse) search, and built-in re-ranking.
Re-ranking matters: a first-pass vector search over-retrieves broadly, then a cross-encoder re-ranker scores that smaller candidate set more precisely - this consistently improves precision over vector similarity alone.
Failure modes are easy to miss without evaluation: confidently-wrong semantic matches, chunking that destroys meaning (especially with too little overlap), and silent embedding-quality drift across model versions if the index isn't regenerated consistently.

Observability & Tracing (Langfuse)

Instruments a request end-to-end: each retrieval call, generation, and parsing step becomes a span carrying latency, token counts, and cost, making the reasoning path (not just the final output) inspectable after the fact.
This is what makes debugging an LLM chain different from debugging a REST API - the "bug" is often in the prompt, context window, or retrieved data, not the code.
Enables direct bottleneck analysis: per-span timing shows whether a slow response is dominated by retrieval, an oversized context window, or a specific model in a fallback chain.

Evaluation, Scoring & A/B Testing

Workflow: sample production traces (or curate a hand-built set) into a dataset, define scoring functions - automated (embedding similarity, LLM-as-judge rubric, exact-match on structured fields) or human (thumbs-up/down) - then run new prompt/model candidates against that dataset before deploying.
A/B testing becomes tractable specifically because the gateway makes model swapping a config change and observability makes quality measurable: route a traffic percentage to each variant, compare Langfuse-recorded scores/latency/cost.
Decision discipline: define the target metric (resolution accuracy, escalation rate, cost/request, latency percentile) before the test, and use a sample large enough to be statistically meaningful - skipping this causes regressions that a proper eval set would have caught.

Fine-Tuning (Last-Resort Layer)

Deliberately sequenced last because it's expensive relative to prompting and RAG, and only pays off once cheaper levers (retrieval quality, prompt design, schema enforcement) are exhausted.
Good candidates: enforcing rigid output formats beyond what JSON-schema constraints guarantee, adapting a smaller/cheaper model to a narrow domain vocabulary, or fixing systematic tone/style failures prompting can't suppress.
Must be validated against the same eval dataset and baseline used for prompt/model A/B tests - without that, a fine-tuning decision is as much a guess as an unmeasured prompt swap.

Trade-offs / Caveats

Every added layer is a real cost: the gateway is a network hop and a third-party uptime/pricing dependency; orchestration frameworks add abstraction overhead and a learning curve that isn't justified for simple single-call use cases.
Structured-output enforcement can behave inconsistently across models fronted by the same gateway - never trust provider-side validation alone.
Retrieval quality degrades silently (embedding drift, bad chunking) without dedicated evaluation catching it.
Observability is easy to under-invest in because it produces no visible feature - but skipping it means losing exactly the historical trace data needed to diagnose the first real incident.

Example in Practice

A Next.js API route composes all five layers per request: it opens a Langfuse trace, calls a retrieval module (vector DB lookup) inside a "retrieval" span, passes the retrieved context plus the question through a gateway call with tiered model fallback (callWithFallback("balanced", ...)), wraps that in a "generation" span, and returns the answer - with the trace flushed and errors logged at the end regardless of outcome. Each concern (retrieval, gateway routing, tracing) lives in its own module, independently testable and swappable, and the route itself contains only composition and error handling.

Paul Serban