AI Engineering vs ML Engineering: How Foundation Models Are Reshaping the Production Pipeline

Introduction

Building software on top of machine learning models is not a new discipline. Recommendation engines, fraud detection systems, and churn prediction models have been running in production for well over a decade, powered by teams of ML engineers who trained, validated, and deployed custom models tailored to narrow, well-defined tasks. The arrival of large-scale foundation models - GPT-4, Claude, Llama, Gemini, and their successors - has not replaced this discipline so much as it has forked it. A new practice, commonly called AI engineering, has emerged to describe the work of building applications on top of models that someone else has already trained.

This distinction matters because the two disciplines, while related, optimize for different problems. Classical ML engineering is fundamentally about producing a model: collecting labeled data, engineering features, selecting an architecture, training, and validating against a held-out set. AI engineering, by contrast, starts from the assumption that a capable general-purpose model already exists, and the engineering challenge is adaptation - prompting, retrieval, fine-tuning, and orchestration - rather than training from scratch. Understanding where these pipelines diverge, and where they still share DNA, is essential for any engineer or technical leader deciding how to structure a team, a data strategy, or a production architecture in 2024 and beyond.

Why This Distinction Matters Now

For most of the last decade, "building an ML-powered feature" meant assembling a pipeline: ingest data, engineer features, train a model (often a gradient-boosted tree or a small neural network), evaluate against metrics like AUC or F1, and deploy behind an API. The team doing this work needed strong statistics and data engineering skills, and the primary bottleneck was almost always data - getting enough labeled examples, cleaning them, and keeping them fresh as the underlying distribution shifted.

Foundation models change the starting point. Instead of beginning with a blank model and a labeled dataset, teams now begin with a pretrained model that already has broad world knowledge and general reasoning ability, exposed through an API or a set of open weights. The work shifts from "how do I get a model to learn this task" to "how do I get this model to perform this task well, reliably, and cheaply". That reframing has consequences that ripple through every stage of the pipeline - from data collection, to development workflow, to how success is even measured.

Because these foundation models are trained on general-purpose corpora, they are capable of open-ended generation rather than fixed-class prediction. A churn model outputs a probability between 0 and 1; a foundation model answering a customer support query can produce an almost unbounded range of valid and invalid responses. This single property - open-endedness - is responsible for a large share of the new engineering challenges discussed throughout this article, particularly around evaluation.

Deep Technical Explanation: Three Structural Shifts

From Model Training to Model Adaptation

The most visible shift is that AI engineers rarely train models from scratch. Where a classical ML engineer might spend weeks iterating on architecture choices - comparing XGBoost against a feedforward network, tuning hyperparameters with cross-validation, or designing a custom loss function - an AI engineer typically starts from a model that is already trained and spends their time on adaptation techniques instead. These include prompt engineering, retrieval-augmented generation (RAG), few-shot examples, and parameter-efficient fine-tuning methods like LoRA (Low-Rank Adaptation). The underlying model's weights are either left untouched or adjusted only partially, which means the skill set shifts from deep statistical modeling toward systems design, prompt design, and data curation for adaptation rather than training.

This does not mean training skills become irrelevant. Organizations with strong data advantages still fine-tune open-weight models, and some build smaller specialized models for narrow, high-volume tasks where a foundation model's latency or cost is prohibitive. But the default starting point has inverted: instead of "we need a model, let's build one," the default is now "there's probably a model that already does most of what we need, let's adapt it".

Bigger Models, Heavier Compute Pressure

The second shift is scale. Classical ML models used in production - logistic regression, gradient-boosted trees, small neural networks - are often small enough to run inference in single-digit milliseconds on a CPU. Foundation models, by contrast, can have tens or hundreds of billions of parameters, and even the smaller distilled variants are orders of magnitude larger than a typical fraud-detection model. This has direct engineering consequences: inference latency becomes a first-class design constraint, token-based pricing from API providers turns every prompt into a cost decision, and techniques like caching, batching, model routing (sending easy queries to a cheap model and hard ones to a capable one), and quantization become central to the architecture rather than optional optimizations.

Open-Ended Outputs and the Evaluation Problem

The third and arguably most consequential shift is output structure. A classical classifier has a fixed, enumerable output space - a class label, a probability, a ranked list. Evaluation is comparatively straightforward: compute precision, recall, AUC-ROC, or RMSE against ground truth labels, and the metric tells you, unambiguously, how good the model is. Foundation models, especially generative ones, produce free-form text, code, or structured JSON with effectively infinite possible outputs. Two different responses to the same prompt might both be "correct" despite sharing almost no surface-level similarity, which makes exact-match or even similarity-based metrics far less reliable.

This is why evaluation - not training - has become one of the central engineering problems in AI engineering. Teams now build evaluation pipelines using techniques such as LLM-as-judge (using a strong model to grade another model's output against a rubric), human preference labeling, and reference-free metrics, often orchestrated through frameworks such as Ragas, DeepEval, or custom harnesses built on top of evaluation libraries like OpenAI's evals framework. None of these approaches is as clean as computing an F1 score, and all require significant engineering investment to make reliable and reproducible.

Data Requirements: What Changes and What Doesn't

Data remains central to both disciplines, but what "data" means and how it's used diverges substantially. In classical ML engineering, the dominant data need is labeled training data: thousands to millions of examples with ground-truth labels, carefully split into train, validation, and test sets, with significant effort spent on label quality, class balance, and avoiding leakage between splits. The data pipeline is often the single largest engineering investment in the entire system, frequently dwarfing the modeling code itself.

In AI engineering, large-scale labeled training data is usually unnecessary, because the foundation model has already absorbed general knowledge during pretraining. Instead, data needs shift toward smaller, higher-quality artifacts: a few dozen to a few hundred well-chosen few-shot examples, a curated knowledge base for retrieval augmentation, a modest fine-tuning dataset for domain adaptation, and - critically - evaluation data. Ironically, teams building on foundation models often need to invest more heavily in building a good evaluation dataset (prompts paired with graded or reference outputs) than in building a training dataset, since the base model's general capability substitutes for most of what training data would otherwise provide.

One data requirement that persists unchanged across both paradigms is the need for production monitoring data - logging real inputs and outputs to detect drift, regressions, and edge cases the team didn't anticipate. Whether the system is a gradient-boosted churn model or a RAG-based support assistant, the discipline of closing the loop between production behavior and the next development cycle remains a core engineering responsibility, not an optional afterthought.

Implementation: Two Pipelines Side by Side

To make the contrast concrete, consider a churn-prediction use case implemented the classical way, and a customer-support triage use case implemented the foundation-model way. Both solve a classification-like problem, but the engineering shape of each is markedly different.

A classical ML pipeline for churn prediction typically looks like a sequence of explicit, versioned steps: feature extraction from a data warehouse, a train/test split, model fitting, and metric evaluation, often wrapped in a reproducible pipeline tool such as scikit-learn's Pipeline or an orchestrator like Airflow or Kubeflow.

from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import roc_auc_score, classification_report

# Feature matrix X and churn labels y come from a feature store
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

pipeline = Pipeline(steps=[
    ("scaler", StandardScaler()),
    ("classifier", GradientBoostingClassifier(
        n_estimators=300,
        max_depth=4,
        learning_rate=0.05,
    )),
])

pipeline.fit(X_train, y_train)

probs = pipeline.predict_proba(X_test)[:, 1]
print("AUC:", roc_auc_score(y_test, probs))
print(classification_report(y_test, probs > 0.5))

Every part of this pipeline is deterministic and reproducible given the same data and random seed. The evaluation metric (AUC) has a single, unambiguous definition, and the model can be retrained on a schedule as new labeled data arrives.

A foundation-model-powered support triage system, by contrast, has no training step at all in its minimal form. The engineering effort concentrates on prompt construction, structured output parsing, and - critically - an evaluation harness that must itself be built rather than imported from a statistics library.

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

interface TriageResult {
  category: "billing" | "technical" | "account" | "other";
  priority: "low" | "medium" | "high";
  reasoning: string;
}

async function triageTicket(ticketText: string): Promise<TriageResult> {
  const response = await client.messages.create({
    model: "claude-sonnet-4-5",
    max_tokens: 300,
    system:
      "You are a support ticket triage assistant. Classify the ticket " +
      "into a category and priority, and return ONLY valid JSON matching " +
      "the schema: {category, priority, reasoning}".,
    messages: [{ role: "user", content: ticketText }],
  });

  const textBlock = response.content.find((b) => b.type === "text");
  if (!textBlock || textBlock.type !== "text") {
    throw new Error("Unexpected response shape from model");
  }

  return JSON.parse(textBlock.text) as TriageResult;
}

// Evaluation harness: compare model output against a small
// human-graded reference set, logging disagreements for review.
async function evaluateAgainstReferenceSet(
  referenceCases: { ticketText: string; expectedCategory: string }[]
) {
  let correct = 0;
  for (const testCase of referenceCases) {
    const result = await triageTicket(testCase.ticketText);
    if (result.category === testCase.expectedCategory) correct++;
  }
  return correct / referenceCases.length;
}

Notice what is absent: there is no training loop, no gradient updates, and no feature engineering in the classical sense. What has appeared instead is prompt and schema design, output parsing with error handling for malformed JSON, and a lightweight evaluation function that, in a real system, would need to be far more sophisticated - likely incorporating an LLM-as-judge step or human review for the "reasoning" field, since correctness of the free-text explanation can't be checked with simple equality.

Trade-offs and Common Pitfalls

Neither pipeline is strictly superior; each carries trade-offs that teams frequently underestimate. Classical ML pipelines offer predictability: given fixed inputs, a deployed model produces the same output every time, latency is low and well-characterized, and the cost per inference is typically negligible compared to a foundation model API call. The cost of this predictability is inflexibility - a churn model trained for one product line usually cannot be repurposed for a different task without retraining, and building the labeled dataset required to reach production quality can take months.

Foundation-model-powered applications offer the opposite trade-off. A single model can be repurposed across dozens of tasks simply by changing the prompt, which dramatically lowers the time-to-first-prototype. The pitfall teams run into most often is treating that speed to prototype as equivalent to speed to production. A prompt that works well on ten manually tested examples frequently breaks down on the long tail of real user inputs, and because outputs are non-deterministic (even at low temperature settings) and difficult to evaluate automatically, regressions can slip into production undetected for far longer than they would in a classical ML system with a hard numeric metric.

A second common pitfall is underestimating cost and latency at scale. It is easy to prototype a feature using a single API call per request and only discover during a scale-up that per-request costs or P99 latency make the feature uneconomical or too slow for its intended use case. Classical ML engineers rarely faced this problem because inference for a small model is close to free; AI engineers must treat compute cost as a design constraint from day one, not an afterthought discovered in a cost review.

A third pitfall, specific to AI engineering, is conflating prompt engineering with system design. Because a well-crafted prompt can produce impressive demo results, teams sometimes skip building retrieval infrastructure, output validation, guardrails, and monitoring - the same production rigor that was considered non-negotiable in classical ML engineering. Foundation models lower the barrier to a working prototype, but they do not lower the barrier to a reliable production system.

Best Practices for Bridging Both Worlds

Teams that succeed at AI engineering tend to borrow the production discipline of classical ML engineering rather than discarding it. This means treating prompts, retrieval corpora, and fine-tuning datasets with the same version control and testing rigor that ML teams already apply to training data and feature pipelines - a prompt change should go through code review and regression testing against an evaluation set, just as a feature change would in a classical pipeline.

It also means investing early in evaluation infrastructure rather than treating it as a nice-to-have. Building even a modest evaluation set of 50–100 representative cases, paired with a mix of automated checks (schema validation, keyword presence, LLM-as-judge scoring) and periodic human review, gives teams a much earlier signal about regressions than relying on user complaints. This evaluation set should grow over time, incorporating real production failures as they're discovered - mirroring the classical ML practice of continuously enriching a training set with hard negatives.

Analogies and Mental Models

One useful mental model is to think of a classical ML model as a custom-tailored suit and a foundation model as a well-stocked department store. Building the custom suit requires measuring the customer precisely (collecting labeled data) and having it made to exact specifications (training), which takes time but fits perfectly for that one person and purpose. Shopping at the department store is faster - you can walk in and leave with something wearable in minutes - but you're now responsible for finding the right fit among thousands of options (prompt and retrieval design) and tailoring it just enough to suit your specific need (fine-tuning or RAG), rather than having built it from nothing.

Another helpful framing is to think of evaluation as the new "ground truth problem". In classical ML, the hard problem was getting enough labeled ground truth to train a model. In AI engineering, the hard problem has moved downstream: ground truth is now needed to evaluate a model you didn't train, and because the outputs are open-ended, defining what "correct" even means requires as much engineering thought as the original labeling effort did in classical ML.

The 80/20 Insight

If there is one idea that captures most of the practical difference between these two disciplines, it's this: classical ML engineering concentrates effort at the start of the pipeline (data and training), while AI engineering concentrates effort at the end of the pipeline (adaptation and evaluation). Internalizing this single shift explains most of the downstream differences in team structure, tooling, and failure modes discussed throughout this article - and it's the first thing to communicate to a classical ML team transitioning into AI engineering work, since their instinct to front-load effort into data collection will otherwise persist past the point where it's useful.

A second, smaller but high-leverage insight is that cost and latency should be modeled as architectural constraints from the first design review, not measured after the fact. Decisions like model routing (using a cheap model for easy cases and an expensive one only when needed), caching repeated queries, and setting token budgets per request have an outsized effect on whether a foundation-model application is viable at scale, and retrofitting them after launch is far more expensive than designing for them upfront.

Key Takeaways

• Shift your mental model from "train a model" to "adapt and evaluate a model". Most of your engineering time should go into prompt design, retrieval, and evaluation harnesses rather than training loops.
• Build an evaluation dataset before you need one. Even 50–100 representative, human-reviewed examples will catch more regressions than manual spot-checking.
• Treat cost and latency as design constraints, not metrics to check later. Model routing, caching, and token budgets belong in the initial architecture.
• Don't discard classical ML production discipline. Version control, regression testing, and monitoring are just as necessary for prompts and retrieval corpora as they were for features and training data.
• Invest in structured output validation. Since foundation models produce open-ended text, always validate and gracefully handle malformed or unexpected outputs before they reach downstream systems.

Conclusion

Classical ML engineering and AI engineering are siblings, not strangers - they share a commitment to data quality, reproducibility, and production monitoring. But the center of gravity has moved. Where ML engineering concentrated its hardest problems in data collection and model training, AI engineering concentrates them in adaptation and evaluation, driven by the fact that foundation models arrive pretrained, operate at far larger scale, and produce open-ended outputs that resist simple metrics.

For engineering leaders, the practical implication is straightforward: don't assume your existing ML playbook transfers unchanged, but don't throw it away either. The teams that build reliable foundation-model applications are usually the ones that keep the production rigor of classical ML engineering - versioning, testing, monitoring - while building genuinely new muscle around prompt engineering, retrieval system design, and the much harder problem of evaluating a model whose correct answer was never a fixed label to begin with.

References

1. Huyen, Chip. AI Engineering: Building Applications with Foundation Models. O'Reilly Media, 2024.
2. Sculley, D. et al. "Hidden Technical Debt in Machine Learning Systems". NeurIPS, 2015.
3. Google. "Rules of Machine Learning: Best Practices for ML Engineering". Google Developers Documentation.
4. Hu, Edward J. et al. "LoRA: Low-Rank Adaptation of Large Language Models". arXiv:2106.09685, 2021.
5. Lewis, Patrick et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks". NeurIPS, 2020.
6. OpenAI. "Evals: Framework for Evaluating LLMs and LLM Systems". GitHub documentation, OpenAI.
7. Ragas Documentation. "Evaluation Framework for Retrieval Augmented Generation Pipelines". docs.ragas.io.
8. Anthropic. "Claude API Documentation". docs.anthropic.com.
9. scikit-learn Developers. "Pipeline and Composite Estimators". scikit-learn.org documentation.
10. Huyen, Chip. Designing Machine Learning Systems. O'Reilly Media, 2022.