Introduction
Ask a classical ML engineer what "good training data" looks like, and they'll usually describe a table: rows of examples, a column of features, and a column with the correct answer - "spam" or "not spam," "fraud" or "not fraud," a number between zero and one. Ask an AI engineer the same question today, and the answer gets considerably murkier, because the "correct answer" to an open-ended prompt like "write a product description for this listing" isn't a single value sitting in a column - it's one of many plausible, well-written responses.
This difference sits at the heart of why dataset engineering looks so different in AI engineering compared to classical ML engineering. It isn't just that the data is bigger or the models are newer; the fundamental nature of what counts as "an answer" has changed, which ripples into how data gets annotated, how it gets cleaned, and how much of it you actually need. Understanding these differences is essential for any engineer deciding how to invest limited time and budget in building or curating a dataset for an AI application.
Close-Ended vs. Open-Ended: Why Annotation Got Harder
Most classical ML use cases are close-ended: a model's output can only be one of a small, predefined set of values. Spam classification is a clean example - an email is either "spam" or "not spam," full stop. This constraint makes annotation comparatively tractable. A labeler looking at an email doesn't need judgment calls about tone, completeness, or creativity; they just need to recognize spam when they see it, a task most people can do quickly and with high agreement between different annotators.
Foundation models, by contrast, are overwhelmingly open-ended. A single prompt like "summarize this article" or "draft a reply to this customer complaint" doesn't have one correct output - it has a wide space of acceptable outputs that vary in tone, length, and phrasing, alongside an equally wide space of unacceptable ones. Annotating open-ended queries is fundamentally harder than annotating close-ended ones, and the gap in difficulty is not small: determining whether an email is spam takes seconds and requires no special skill, while writing a high-quality essay response, or judging whether someone else's essay response is good, requires time, subject-matter familiarity, and often disagreement-prone subjective judgment. This is why data annotation has become a much bigger challenge in AI engineering than it ever was in classical ML - not because annotation itself is a new activity, but because the task being annotated got dramatically harder to annotate consistently.
This difficulty compounds when annotation moves from grading a single response to comparing multiple responses, which is exactly what techniques like Reinforcement Learning from Human Feedback (RLHF) require. Instead of asking "is this correct," annotators are asked "which of these two responses is better," a comparative judgment that still requires expertise and is still prone to disagreement, but that at least sidesteps the need to specify one single gold-standard answer for an open-ended task.
From Tabular Data to Unstructured Data
A second structural difference lies in the shape of the data itself. Classical ML engineering works predominantly with tabular data: structured rows and columns representing features like transaction amount, customer tenure, or pixel values arranged into a fixed-size array. This structure lends itself naturally to well-established preprocessing techniques - normalization, one-hot encoding, handling missing values in a specific column - all of which assume a known, fixed schema.
Foundation models work with unstructured data: raw text, code, images, and increasingly audio and video, none of which arrive in a predefined column-based schema. This shift changes what "data manipulation" even means. Instead of imputing missing numeric values or scaling a feature column, AI engineering's data manipulation work centers on operations like deduplication (removing repeated or near-identical documents that would otherwise bias training or pollute a retrieval index), tokenization (converting raw text into the discrete units a model actually consumes), and context retrieval (finding and assembling the right unstructured passages to feed into a model at inference time). Quality control takes on a different character too - rather than checking whether a numeric feature falls within an expected range, AI engineering quality control often means removing personally identifiable information, filtering toxic or harmful content, and screening a corpus for copyright or licensing issues before it's used for training or retrieval.
These operations require different tooling than classical ML data pipelines. Deduplication at the scale of a large text corpus typically relies on techniques like MinHash or locality-sensitive hashing to detect near-duplicate documents efficiently, since exact string matching misses paraphrased or lightly edited duplicates. Tokenization relies on subword algorithms such as byte-pair encoding (BPE) or SentencePiece, which split text into a vocabulary of frequent subword units rather than whole words, balancing vocabulary size against the ability to represent rare or unseen words. None of this tooling has an equivalent in a classical tabular pipeline built around pandas and scikit-learn preprocessing, because the underlying data has a fundamentally different shape.
Implementation: Data Preparation in Practice
To make this concrete, consider what a realistic data preparation step looks like in each paradigm. A classical ML pipeline working with tabular data focuses on consistent schema handling and numeric transformations.
import pandas as pd
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
transactions = pd.read_csv("transactions.csv")
numeric_features = ["amount", "account_age_days"]
categorical_features = ["merchant_category", "payment_method"]
preprocessor = ColumnTransformer(transformers=[
("num", Pipeline(steps=[
("imputer", SimpleImputer(strategy="median")),
("scaler", StandardScaler()),
]), numeric_features),
("cat", OneHotEncoder(handle_unknown="ignore"), categorical_features),
])
processed = preprocessor.fit_transform(transactions)
Every step here assumes a fixed, known schema: specific numeric columns to scale, specific categorical columns to encode. The data manipulation problem is well-defined because the data itself is well-defined.
Preparing unstructured text data for a retrieval-augmented AI application looks structurally different. Instead of column-based transformations, the pipeline deduplicates documents, chunks and tokenizes them, and filters out content that shouldn't be retrievable or trainable on.
from datasketch import MinHash, MinHashLSH
import re
def is_pii_or_toxic(text: str) -> bool:
"""Simplified placeholder for PII/toxicity screening."""
pii_patterns = [r"\b\d{3}-\d{2}-\d{4}\b", r"\b[\w.+-]+@[\w-]+\.\w+\b"]
return any(re.search(pattern, text) for pattern in pii_patterns)
def deduplicate_documents(documents: list[str], threshold: float = 0.8) -> list[str]:
lsh = MinHashLSH(threshold=threshold, num_perm=128)
unique_docs = []
for i, doc in enumerate(documents):
minhash = MinHash(num_perm=128)
for word in doc.split():
minhash.update(word.encode("utf8"))
if not lsh.query(minhash):
lsh.insert(str(i), minhash)
unique_docs.append(doc)
return unique_docs
def prepare_corpus(raw_documents: list[str]) -> list[str]:
filtered = [doc for doc in raw_documents if not is_pii_or_toxic(doc)]
deduplicated = deduplicate_documents(filtered)
return deduplicated
Note the complete absence of columns, numeric scaling, or categorical encoding. The operations here - near-duplicate detection via MinHash, content screening for sensitive information - have no direct counterpart in the tabular pipeline, because the underlying problem being solved (cleaning a large, unstructured corpus) didn't exist in the same form in classical ML data engineering.
How Much Data You Actually Need
One of the most practically important differences between classical ML and AI engineering is how the required data volume scales with the adaptation technique chosen. Training a model from scratch demands the most data by a wide margin - this is the regime that produced datasets like ImageNet's millions of labeled images, and it's the regime foundation model pretraining operates in at an even larger scale, consuming web-scale text corpora. Fine-tuning an already-pretrained foundation model requires substantially less data than training from scratch, because the model already has broad capability and only needs adjustment toward a specific task or style - fine-tuning sets in practice often range from hundreds to a few thousand examples, rather than millions. Prompt engineering, the lightest-weight adaptation technique, often requires no dedicated training dataset at all, relying instead on a handful of well-chosen few-shot examples embedded directly in the prompt, sometimes as few as ten.
This spectrum has a direct, practical consequence for time-to-market. Adapting an existing powerful foundation model to a new task is generally far easier than building a model for that task from the ground up - the difference between spending a weekend writing and testing prompts against ten examples, versus spending six months collecting and labeling a million examples to train a model from scratch. This isn't a minor convenience; it's the core economic shift that foundation models introduced. Teams that previously needed a multi-month data collection effort before they could ship anything can now validate a product idea using an off-the-shelf model and a small, carefully chosen evaluation set, reserving the heavier data investment for later, if and when it's actually justified by evidence that a simpler approach has hit its ceiling.
Trade-offs and Pitfalls in Dataset Engineering
The reduced data requirements of adaptation techniques create a trap that's easy to fall into: treating a small, convenient dataset as sufficient simply because the adaptation technique technically works with less data. A fine-tuning set of a few hundred examples can produce a model that performs well on inputs similar to those examples while failing badly on the long tail of real-world variation it never saw. Because open-ended tasks have such a wide space of valid and invalid outputs, a small dataset that looks adequate on a handful of manual spot-checks can still miss entire categories of failure that only show up once the application faces diverse production traffic.
A second pitfall is underestimating the ongoing cost of data quality control for unstructured data. Classical tabular pipelines can validate data quality with relatively simple, deterministic checks - is this value within range, is this column non-null. Unstructured data quality control is messier and more judgment-dependent: toxicity and PII filters produce false positives and false negatives, deduplication thresholds trade off between removing genuine duplicates and discarding merely similar-but-distinct content, and content licensing concerns require ongoing vigilance rather than a one-time check. Teams that treat these filters as a "set it and forget it" step often discover problems only after a production incident, rather than catching them during data preparation.
A third, often-cited claim worth examining critically is that because foundation models are increasingly commoditized, data becomes the primary differentiator between competing applications. This is directionally true but easy to overstate: data quality matters enormously, but so does the engineering quality of evaluation, context construction, and interface design layered on top of that data. Teams that invest heavily in dataset engineering while neglecting evaluation rigor often end up with a dataset they believe is high-quality but have no reliable way to verify actually improves the application's real-world performance.
Best Practices for Dataset Engineering in AI Applications
Match your data investment to your adaptation technique rather than defaulting to "more data is always better." If prompt engineering with a handful of few-shot examples solves the problem adequately, there's little value in prematurely building a large fine-tuning dataset - that investment should wait until production evidence shows a specific, recurring gap that prompting alone can't close. Conversely, once fine-tuning is justified, invest in a dataset that reflects the genuine diversity of production inputs, not just the easy, common cases that are simplest to collect.
Build deduplication, PII filtering, and toxicity screening into your data pipeline as standard, versioned steps rather than ad hoc scripts run once. Because these filters inevitably have false positive and false negative rates, treat their thresholds as configuration that gets reviewed and tuned over time, the same way a classical ML team would monitor and retune a fraud detection threshold. For annotation of open-ended outputs, prefer comparative judgments (which response is better) over absolute judgments (is this response correct) wherever possible, since comparative judgments are typically faster for annotators to make consistently and map naturally onto techniques like RLHF.
80/20 Insight
If there's one idea that captures most of the practical difference in dataset engineering, it's this: the hardest part of data work in AI engineering isn't collecting more data, it's judging the quality of open-ended outputs consistently. Classical ML struggled with the cost of getting enough labels; AI engineering struggles with the difficulty of defining what a "good" label even looks like for a task that doesn't have one right answer. Teams that build a disciplined, consistent process for quality judgment - whether through clear rubrics, comparative annotation, or LLM-as-judge scoring - get disproportionately more value from their dataset engineering effort than teams that simply try to collect larger volumes of unstructured data.
Key Takeaways
- Open-ended tasks make annotation fundamentally harder than classical close-ended classification, because there's no single correct answer to check against.
- Dataset engineering for foundation models centers on unstructured data operations - deduplication, tokenization, context retrieval, and quality filtering - rather than tabular transformations.
- Data volume requirements scale with your adaptation technique: training from scratch needs the most data, fine-tuning needs substantially less, and prompt engineering often needs only a handful of examples.
- A small fine-tuning dataset that looks adequate on manual review can still miss significant failure modes in production, especially for open-ended tasks with a wide space of valid and invalid outputs.
- Invest in consistent quality judgment processes - rubrics, comparative annotation, or LLM-as-judge - since defining "good" is the harder problem in AI engineering, not collecting raw volume.
Conclusion
Dataset engineering didn't become less important with the arrival of foundation models - it became differently important. Classical ML engineering's challenges centered on the cost and scale of collecting enough labeled, tabular examples to train a model from scratch. AI engineering's challenges center on the much harder problem of judging quality for open-ended outputs, cleaning unstructured data at scale, and calibrating exactly how much data a given adaptation technique actually requires.
For engineering teams, the practical takeaway is to treat data investment as a dial that should be tuned to the adaptation technique in use, not a fixed cost paid upfront regardless of approach. Starting with prompt engineering and a small, carefully curated evaluation set, then moving to fine-tuning only once production evidence justifies it, captures most of the economic advantage foundation models offer - while still leaving room to invest seriously in dataset quality and annotation rigor once the data actually matters for the task at hand.
References
- Huyen, Chip. AI Engineering: Building Applications with Foundation Models. O'Reilly Media, 2024.
- Huyen, Chip. Designing Machine Learning Systems. O'Reilly Media, 2022.
- Ouyang, Long et al. "Training Language Models to Follow Instructions with Human Feedback." NeurIPS, 2022. (RLHF and comparative annotation)
- Sennrich, Rico, Haddow, Barry, and Birch, Alexandra. "Neural Machine Translation of Rare Words with Subword Units." ACL, 2016. (Byte-pair encoding)
- Kudo, Taku, and Richardson, John. "SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing." EMNLP, 2018.
- Broder, Andrei Z. "On the Resemblance and Containment of Documents." Proceedings of Compression and Complexity of Sequences, 1997. (MinHash for near-duplicate detection)
- Deng, Jia et al. "ImageNet: A Large-Scale Hierarchical Image Database." CVPR, 2009.
- scikit-learn Developers. "Preprocessing Data." scikit-learn.org documentation.
- Hu, Edward J. et al. "LoRA: Low-Rank Adaptation of Large Language Models." arXiv:2106.09685, 2021.