Introduction
Every AI application, whether it's a fraud-detection model from 2015 or a retrieval-augmented chatbot built this year, rests on the same three structural layers: application development, model development, and infrastructure. What has changed is not the existence of these layers but their relative size, their order of construction, and where a team's competitive advantage now lives. Before foundation models, most engineering teams lived almost entirely in the bottom two layers - gathering data, training models, and building the infrastructure to serve them - with the application layer treated as a thin wrapper around a prediction endpoint.
Foundation models inverted this balance. Because a capable, general-purpose model is now available via API or open weights, the application layer has absorbed the majority of new engineering effort, while the infrastructure layer has stayed comparatively stable. Understanding this three-layer stack - and precisely which parts of it moved, which stayed fixed, and why - gives engineers and technical leaders a clear mental map for deciding where to invest, hire, and build tooling.
The Three-Layer Stack: A Structural Overview
At the top sits application development: the layer responsible for prompt engineering, context construction (assembling the right information for a model to reason over, often via retrieval), evaluation, and interface design. This is the layer that has seen the most activity since foundation models emerged, precisely because it's now the layer accessible to the widest range of engineers - you no longer need a PhD in machine learning to build something useful, just a good understanding of how to structure a prompt and evaluate its output reliably.
In the middle sits model development: the tooling and processes for modeling, training, fine-tuning, and inference optimization. Because data is central to producing or adapting a model, this layer also encompasses dataset engineering - collecting, cleaning, and curating the examples used for pretraining, fine-tuning, or building a retrieval corpus. Model development still demands rigorous evaluation, just as application development does, though the object being evaluated differs: here it's the model's raw capability, not the end-to-end application behavior.
At the bottom sits infrastructure: tooling for model serving, data and compute management, and monitoring. This layer is the most structurally conservative of the three. Whether a team is serving a gradient-boosted tree or a 70-billion-parameter language model, the underlying problems - how do you route requests to a model efficiently, how do you track compute costs, how do you detect when something in production has silently broken - remain largely the same in shape, even though the specific tools (GPU orchestration, vector databases, token-based cost tracking) have evolved to match the new workload.
Why the Stack Shifted: Context and Problem Overview
To understand why effort redistributed toward the top layer, it helps to recall what the stack looked like before foundation models were widely accessible. A team building a churn-prediction model or a product recommender had no choice but to start at the bottom two layers: assemble a labeled dataset, choose and train a model architecture, and only then build a thin application layer to serve predictions to end users. Model development wasn't optional - it was the entire point of the project, and the quality of that proprietary model was often the team's primary competitive advantage.
Foundation models changed the starting conditions. A team today can call an API from OpenAI, Anthropic, or Google, or download open weights from Meta's Llama family, and immediately have access to a model with broad reasoning and language capability that would have taken years and enormous compute budgets to train independently. This means a huge number of teams now start their AI initiatives by working almost exclusively in the application layer - writing prompts, constructing context through retrieval, and building evaluation harnesses - without ever touching the model development layer at all. Model development hasn't disappeared, but it's become something teams invest in selectively, once they have evidence that a prompt-only or retrieval-augmented approach has hit its ceiling.
This shift also explains why infrastructure investment has remained comparatively stable in its purpose, even as its implementation has changed. The need to manage compute, monitor system health, and serve requests reliably didn't go away - it was simply inherited by a new set of components. A monitoring system that once tracked prediction drift on a classifier's output distribution now tracks token usage, latency percentiles, and the rate of malformed JSON responses from an LLM. The job title of the layer stayed the same; the instruments changed.
Deep Technical Explanation: What Each Layer Actually Contains
Application Development: Prompting, Context, and Evaluation
Application development in the foundation-model era is built around four interlocking components: prompt engineering, context construction, evaluation, and interface design. Prompt engineering is the practice of crafting instructions and examples that reliably steer a model's behavior - system prompts, few-shot examples, output format constraints. Context construction goes a step further: it's the engineering work of assembling the right information for the model to reason over at inference time, typically through retrieval-augmented generation (RAG), tool calling, or structured context windows pulled from a knowledge base or live data source.
Evaluation sits at the center of this layer because, without a fixed label to check against, the only way to know whether an application is actually working is to build a deliberate evaluation process - a mix of automated checks, LLM-as-judge scoring, and human review, run against a representative set of real or synthetic inputs. Interface design, the fourth component, is often underestimated by engineers coming from a classical ML background: a chat interface, a structured form, or a tool-use pattern with visible intermediate steps all shape how forgiving users are of a model's imperfections, and a well-designed interface can make an 80%-accurate application feel reliable while a poorly designed one makes a 95%-accurate application feel broken.
Model Development: Modeling, Training, and Dataset Engineering
Model development didn't disappear with foundation models - it shifted in scope and frequency. Teams with strong proprietary data still fine-tune open-weight models using techniques like LoRA, or train smaller specialized models for narrow, high-volume tasks where an API call's latency or cost is prohibitive. Dataset engineering lives in this layer because any model development effort - pretraining, fine-tuning, or building embeddings for a retrieval index - depends on curated data: cleaning examples, deduplicating, balancing distributions, and in the LLM era, often generating synthetic data to fill gaps. Inference optimization, including quantization, batching, and distillation, also belongs here, since making a model cheaper or faster to run is itself a model-development concern distinct from how an application prompts that model.
Infrastructure: The Stable Foundation
Infrastructure provides the operational substrate the other two layers depend on: compute management (provisioning GPUs or managing API rate limits), data management (storing training sets, embeddings, or logs), model serving (exposing a model behind a low-latency API), and monitoring (tracking latency, error rates, and output quality over time). What makes this layer distinctive is how little its purpose has changed relative to classical ML systems - the tooling has evolved (vector databases like Pinecone or Weaviate for embeddings, GPU orchestration platforms like Ray or Kubernetes-based schedulers), but the underlying responsibility of keeping a model reliably servable and observable in production is unchanged.
Implementation: A Practical Walkthrough Across the Three Layers
Consider a concrete example: a team building an internal documentation search assistant. Following the natural top-down order the stack suggests, they start at the application layer with a simple RAG pipeline - no custom model, no fine-tuning, just prompt engineering and context construction against an existing foundation model.
import Anthropic from "@anthropic-ai/sdk";
import { embedQuery, retrieveTopChunks } from "./retrieval";
const client = new Anthropic();
async function answerDocsQuestion(question: string): Promise<string> {
// Application layer: context construction via retrieval
const queryEmbedding = await embedQuery(question);
const contextChunks = await retrieveTopChunks(queryEmbedding, 5);
const contextBlock = contextChunks
.map((chunk, i) => `[${i + 1}] ${chunk.text}`)
.join("\n\n");
// Application layer: prompt engineering
const response = await client.messages.create({
model: "claude-sonnet-4-5",
max_tokens: 500,
system:
"Answer the user's question using ONLY the provided context. " +
"Cite sources by their bracketed number. If the context doesn't " +
"contain the answer, say so explicitly.",
messages: [
{
role: "user",
content: `Context:\n${contextBlock}\n\nQuestion: ${question}`,
},
],
});
const textBlock = response.content.find((b) => b.type === "text");
return textBlock && textBlock.type === "text" ? textBlock.text : "";
}
This entire implementation touches only the application layer and borrows retrieval infrastructure the team already has. No model was trained, no dataset was labeled from scratch - the retrieval corpus is simply the existing documentation, reindexed as embeddings.
Only after this system is in production and the team notices a recurring failure mode - say, the model consistently struggles with questions about a particular internal API's edge cases - does it make sense to move down into model development. At that point, the team might curate a small fine-tuning dataset from logged production failures and successes, using a structure like this to prepare the data:
import json
def build_finetuning_examples(production_logs: list[dict]) -> list[dict]:
"""
Convert reviewed production Q&A logs into a fine-tuning dataset.
Each log entry has been human-reviewed for correctness.
"""
examples = []
for log in production_logs:
if log["human_reviewed"] and log["label"] == "correct":
examples.append({
"messages": [
{"role": "system", "content": log["system_prompt"]},
{"role": "user", "content": log["question"]},
{"role": "assistant", "content": log["approved_answer"]},
]
})
return examples
def write_jsonl(examples: list[dict], path: str) -> None:
with open(path, "w") as f:
for example in examples:
f.write(json.dumps(example) + "\n")
Note the direction of the build: context and prompt first, dataset curation and fine-tuning only once the simpler approach's limits are understood through production evidence. The infrastructure layer underneath - the vector store, the API client, the logging pipeline - stayed constant throughout both phases.
Trade-offs and Pitfalls Across the Stack
The reordering of effort toward the application layer creates real trade-offs, not just convenience. Because many teams now build on the same handful of foundation models, model quality is no longer a reliable source of differentiation the way a proprietary fraud model once was. A competitor can call the same API you do. This pushes the locus of differentiation up into the application layer - the quality of your evaluation process, the sophistication of your context construction, and the polish of your interface become the things that separate a mediocre product from an excellent one. Teams that treat the application layer as a thin, interchangeable wrapper around a model API, rather than as a genuine source of competitive advantage, tend to build products that are easy for competitors to replicate.
A second pitfall is skipping the model development layer too permanently. Because it's now possible to build an entire working product without ever training or fine-tuning anything, some teams never revisit the question of whether a cheaper, faster, or more specialized model would serve them better at scale. This is fine during early-stage validation, but as usage grows, the economics can shift - a narrowly fine-tuned smaller model can sometimes outperform a general-purpose foundation model on a specific task at a fraction of the inference cost. Teams that never build the organizational muscle to move down into model development when it's warranted end up overpaying indefinitely for a capability they could have specialized away.
A third pitfall sits in the infrastructure layer, and it's more subtle: because the tooling looks different (vector databases instead of feature stores, token metering instead of request counting), some teams rebuild infrastructure practices from scratch rather than recognizing that the underlying discipline - careful monitoring, cost attribution, and capacity planning - already exists in their organization's classical ML or backend infrastructure teams. Reinventing monitoring and serving practices wastes time that could be spent adapting existing infrastructure discipline to the new tooling.
Best Practices for Navigating the Stack
The build order itself is a best practice worth stating explicitly: start at the application layer, prove the product has value using an off-the-shelf foundation model, and only invest in model development once production data reveals a specific, well-defined gap that fine-tuning or custom training would close. This "product first, data and model later" order is a genuine inversion of the traditional ML engineering sequence, and resisting the instinct to front-load data collection or model selection is one of the highest-leverage mindset shifts a classical ML team can make when moving into AI engineering.
It's equally important to keep AI engineers embedded in product decisions rather than walling them off the way model-development teams were often isolated in the classical ML era. Because the application layer is now where most of the differentiating engineering happens, the people doing that work - writing prompts, designing evaluation rubrics, shaping context construction - need direct visibility into user behavior and product goals. An AI engineer who only sees anonymized metrics, without understanding why users ask the questions they ask, will struggle to improve the parts of the system that matter most.
Finally, treat the infrastructure layer as something to extend rather than rebuild. Most organizations already have mature practices around compute provisioning, cost monitoring, and incident response from their classical software and ML systems. The specific metrics change - latency percentiles for token generation instead of batch job completion times, cost per request instead of cost per training run - but the discipline of systematic monitoring and capacity planning transfers directly and shouldn't be reinvented from zero.
Mental Model: The Stack as a Funnel of Investment
A useful way to think about the three-layer stack is as a funnel of increasing investment and decreasing frequency. Every team touches the application layer - it's the cheapest to experiment in and the fastest to iterate on, so naturally the most activity happens there. A smaller number of teams, those with specific data advantages or cost pressures that a general-purpose model can't satisfy, move down into model development. And only a small fraction of organizations - typically those building foundation models themselves or operating at a scale where infrastructure is a genuine competitive lever - invest heavily in building novel infrastructure rather than consuming existing platforms.
This funnel shape also explains the earlier point about where differentiation now lives. Because so many teams occupy the same wide top of the funnel, competing on the same models and often similar prompting techniques, the quality of execution within that layer - evaluation rigor, context construction sophistication, interface design - becomes the differentiator, simply because it's the layer where the most teams are competing for the same underlying capability.
Key Takeaways
- Start at the application layer and move down only when justified. Build with an off-the-shelf foundation model first; invest in fine-tuning or custom training only once production data shows a specific gap.
- Treat evaluation as a first-class component of the application layer, not an afterthought - it's the thing that tells you whether your prompt and context construction are actually working.
- Recognize that differentiation has moved up the stack. When many teams use the same foundation model, your evaluation process, context construction, and interface design are your competitive edge, not the model itself.
- Keep AI engineers embedded in product decisions. The disjointed relationship between model teams and product teams from the classical ML era doesn't serve well when application-layer work is this central to the product.
- Extend existing infrastructure discipline rather than rebuilding it. Compute management, monitoring, and serving principles from classical ML and backend engineering transfer directly, even as the specific tools change.
Conclusion
The three-layer stack - application development, model development, and infrastructure - isn't a new invention of the foundation-model era; it's the same structure that's always underpinned AI systems. What foundation models changed is the center of gravity: application development has absorbed most of the new engineering activity, model development has become a selective, evidence-driven investment rather than a mandatory first step, and infrastructure has quietly adapted its tooling while keeping its core responsibilities intact.
For engineering leaders, the practical implication is to resist two opposite mistakes: treating the application layer as trivial because "anyone can call an API," and treating the stack as so new that classical ML and infrastructure discipline no longer applies. The organizations that build durable AI products are the ones that take the application layer as seriously as the model layer was once taken, while carrying forward the parts of classical engineering practice - systematic evaluation, feedback loops, and infrastructure rigor - that were never really about the model in the first place.
References
- Huyen, Chip. AI Engineering: Building Applications with Foundation Models. O'Reilly Media, 2024.
- Huyen, Chip. Designing Machine Learning Systems. O'Reilly Media, 2022.
- Sculley, D. et al. "Hidden Technical Debt in Machine Learning Systems." NeurIPS, 2015.
- Lewis, Patrick et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." NeurIPS, 2020.
- Hu, Edward J. et al. "LoRA: Low-Rank Adaptation of Large Language Models." arXiv:2106.09685, 2021.
- Anthropic. "Claude API Documentation." docs.anthropic.com.
- Google. "Rules of Machine Learning: Best Practices for ML Engineering." Google Developers Documentation.
- Ray Project Documentation. "Ray for Distributed Compute and Model Serving." docs.ray.io.
- Pinecone. "Vector Database Documentation." docs.pinecone.io.
- OpenAI. "Evals: Framework for Evaluating LLMs and LLM Systems." GitHub documentation, OpenAI.