Self-Supervision vs. Supervision: Why Foundation Models Need Different Data Than Classical ML

Introduction

Every machine learning system, no matter how sophisticated, is ultimately a function of the data it was trained on. For most of the 2010s, that meant labeled data - carefully annotated examples that told a model exactly what the correct answer should be for a given input. Fraud detection models learned from transactions tagged "fraud" or "not fraud." Image classifiers learned from millions of pictures tagged with object names. This approach, called supervised learning, powered the first wave of production ML systems and remains the backbone of countless classical pipelines running today.

Foundation models broke this pattern, not by abandoning the need for data, but by changing what kind of data they need and how that data gets labeled. Instead of requiring a human to annotate every example, large language models learn from raw text by predicting what comes next - a technique called self-supervised learning. This single shift in data requirements is arguably the most consequential technical difference between classical ML pipelines and foundation-model-powered applications, because it's what allowed models to scale from millions of examples to trillions of tokens. Understanding this distinction is essential for any engineer deciding how to source, prepare, or fine-tune data for an AI system.

The Supervision Bottleneck in Classical ML

Classical ML models typically require supervision: training data with explicit labels that teach the model the correct output for each input. A fraud detection model needs examples of transactions, each tagged "fraud" or "not fraud." A churn model needs historical customer records labeled with whether that customer eventually left. Without these labels, standard supervised algorithms - logistic regression, gradient-boosted trees, convolutional neural networks trained for classification - have nothing to learn from, because the entire training signal comes from the gap between the model's prediction and the known correct answer.

This approach defined the breakthrough era of deep learning. AlexNet, the convolutional neural network that is widely credited with igniting the modern deep learning revolution after its strong performance in the 2012 ImageNet Large Scale Visual Recognition Challenge, was a supervised model trained to classify images across the categories in the ImageNet dataset, itself built from millions of hand-labeled images. The success of that era rested heavily on the willingness and ability of teams to invest in large, carefully labeled datasets, often constructed by services or crowdsourced labor using platforms like Amazon Mechanical Turk.

The drawback of this approach is cost and time. Labeling is a human bottleneck, and the price scales directly with the number of examples needed and the difficulty of the labeling task itself. Simple, everyday labeling - identifying whether a photo contains a cat or a dog - can be done quickly and cheaply by almost anyone. But not all labeling tasks are that accessible. Producing accurate Latin translations for training an English-to-Latin model requires someone who is actually fluent in Latin, which is a far smaller and more expensive labor pool than people who can identify household objects. Labeling whether a CT scan shows signs of cancer requires a trained radiologist, and the cost and scarcity of that expertise makes large-scale labeling for specialized medical tasks enormously expensive, if not practically infeasible at the scale modern deep learning models require.

How Self-Supervision Changed the Equation

Foundation models overcome the labeling bottleneck through self-supervised learning, a training paradigm where the model generates its own training signal from unlabeled data rather than relying on human-provided labels. In the context of language models, this works by treating prediction of the next token as the task itself: every sequence of text inherently contains both the input context and the correct output, because the next word in the sequence is simply the next word that's already there.

Consider the sentence "I love street food." This single sentence, tokenized and processed sequentially, yields multiple training examples automatically. Given the special beginning-of-sequence marker, the model learns to predict "I" from <BOS>. Given <BOS>, I, it learns to predict "love." Given <BOS>, I, love, it learns to predict "street," and so on, continuing through "food" and finally the end-of-sequence marker. No human had to sit down and label any of these six training pairs - the labels were already embedded in the raw text, waiting to be extracted by simply shifting the sequence by one position. This is the mechanism underlying causal (autoregressive) language modeling, the training objective used by GPT-style models.

The consequence of this shift is enormous. Because text sequences already exist in vast quantities across books, articles, websites, code repositories, and forums, the constraint that limited classical supervised learning - the cost and time of human labeling - mostly disappears for language model pretraining. A team no longer needs to pay someone to label a million sentences; they need only find a million sentences that already exist, which is a dramatically easier problem. This is precisely what allowed foundation models to scale training data from the millions of labeled examples typical of classical ML to the trillions of tokens used to pretrain today's large language models.

Implementing Both Paradigms: A Practical Comparison

To make this distinction concrete, consider how each paradigm looks in code. A classical supervised fraud detection pipeline requires an explicitly labeled dataset before any training can begin - the labels are a precondition, not a byproduct of the data itself.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report

# Each row requires an explicit human- or system-assigned label
transactions = pd.read_csv("labeled_transactions.csv")
# Columns: amount, merchant_category, time_of_day, ..., is_fraud

X = transactions.drop(columns=["is_fraud"])
y = transactions["is_fraud"]  # The label that had to be collected/assigned

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)

model = RandomForestClassifier(n_estimators=200, max_depth=10)
model.fit(X_train, y_train)

predictions = model.predict(X_test)
print(classification_report(y_test, predictions))

Every row in labeled_transactions.csv had to arrive with an is_fraud label already attached, either from a confirmed fraud investigation, a chargeback record, or manual review - none of which the model can generate on its own.

By contrast, preparing data for self-supervised language model training requires no explicit labeling step at all. The "labels" are derived programmatically from the raw text itself, by constructing input-output pairs through a sliding window over the token sequence.

def build_self_supervised_examples(token_ids: list[int], context_window: int = 5):
    """
    Generate (input_context, target_token) pairs from a raw token sequence.
    No human labeling required - the text itself provides the supervision signal.
    """
    examples = []
    for i in range(1, len(token_ids)):
        start = max(0, i - context_window)
        context = token_ids[start:i]
        target = token_ids[i]
        examples.append({"context": context, "target": target})
    return examples

# Raw, unlabeled text - tokenized but otherwise untouched by human annotators
text = "I love street food."
token_ids = tokenize(text)  # e.g., [BOS_ID, 42, 17, 103, 88, EOS_ID]

training_pairs = build_self_supervised_examples(token_ids)
# Each pair was extracted directly from the sentence structure itself

The structural difference is stark: the fraud detection example depends entirely on an external, human-curated label column that must exist before training can start, while the language modeling example manufactures its own supervision signal from data that required no annotation effort beyond being collected and tokenized.

Trade-offs and Limitations of Each Approach

Self-supervision's scalability advantage comes with real trade-offs that engineering teams need to understand rather than treat as a free lunch. The most significant is that self-supervised pretraining teaches a model general patterns of language - grammar, common knowledge, reasoning patterns embedded in text - but it does not teach the model to perform any specific, well-defined task the way supervised training does. A language model pretrained purely through next-token prediction has no inherent notion of "classify this transaction as fraudulent" the way a supervised fraud model does; it has only learned to continue text plausibly. This is why foundation models typically go through additional stages after pretraining - instruction tuning and reinforcement learning from human feedback (RLHF), as used in training models like InstructGPT - stages that do reintroduce human-labeled or human-preference data, just at a much smaller scale than pretraining.

A second limitation is that self-supervision doesn't eliminate the need for labeled data entirely - it relocates it. Evaluation still typically requires some form of ground truth or human judgment, and fine-tuning a foundation model for a specific downstream task (say, classifying support tickets into categories) often still benefits from a modest labeled dataset, even though that dataset can be orders of magnitude smaller than what a from-scratch supervised model would have required, precisely because the model already has a strong general-purpose starting point from pretraining.

A third consideration, often underappreciated, is data quality and provenance. Because self-supervised pretraining consumes enormous quantities of text scraped from the open web, books, and other sources, it inherits whatever biases, inaccuracies, and inconsistencies exist in that data. Classical supervised datasets, while expensive to build, are at least curated with a specific, well-understood labeling process in mind, giving teams more direct control over what the model is taught. Teams building on foundation models need to be aware that the model's "knowledge" is a byproduct of its training corpus, not a deliberately engineered label set, which has implications for fairness, factual accuracy, and domain specificity.

Best Practices for Working With Each Data Paradigm

When your application genuinely depends on classical supervised learning - for instance, building a narrow, high-precision classifier where a foundation model's cost or latency isn't justified - invest early in a clear labeling specification. Ambiguous labeling guidelines are one of the most common sources of poor model performance in supervised pipelines, because inconsistent labels compound into inconsistent model behavior. Tools like label validation scripts, inter-annotator agreement checks, and a well-documented labeling guide pay for themselves many times over compared to the cost of retraining on a cleaned-up dataset later.

When working with foundation models, the practical data investment shifts toward two areas: curating high-quality fine-tuning or instruction-tuning data in small, carefully reviewed batches, and building retrieval corpora or evaluation datasets rather than massive labeled training sets. Since the heavy lifting of general language understanding is already handled by pretraining, teams get far more leverage from a few hundred well-chosen, high-quality examples for fine-tuning than from attempting to replicate the scale of labeled data used in classical supervised learning. It's also worth treating any data sourced for self-supervised pretraining or retrieval with the same provenance discipline recommended for supervised datasets - tracking where it came from, filtering for quality, and being deliberate about what a model is allowed to learn from, even when no explicit labeling step was required to collect it.

Mental Model: The Library vs. The Lesson Plan

A useful analogy is to think of classical supervised learning as writing a lesson plan: a teacher (the labeler) must decide, for every single example, exactly what the correct answer is before the student (the model) ever sees it. This is thorough and precise, but writing a lesson plan for a million examples takes a proportional amount of teacher time.

Self-supervised learning is closer to handing a student an entire library and asking them to read everything, inferring patterns, grammar, and relationships between ideas without anyone grading each sentence along the way. The student develops broad, general competence this way - but no one ever explicitly taught them to solve a specific exam question. That's why a foundation model, having "read the library," often still needs a short, targeted lesson (fine-tuning or instruction tuning) before it reliably performs a specific task well.

Key Takeaways

• Classical supervised learning requires explicit labels that must be collected or assigned by humans before training can begin, with cost scaling directly with labeling difficulty and dataset size.
• Self-supervised learning extracts its own training signal from raw data, as in next-token prediction for language models, removing the labeling bottleneck for pretraining at scale.
• Self-supervision doesn't eliminate labeled data needs - it relocates them to smaller-scale stages like instruction tuning, RLHF, and evaluation.
• Data quality and provenance matter in both paradigms, but self-supervised pretraining inherits whatever biases and inaccuracies exist in its much larger, less curated source corpus.
• Invest labeling effort where it has the most leverage: large labeling specs for from-scratch supervised models, small high-quality fine-tuning sets for foundation model adaptation.

Conclusion

The shift from supervision to self-supervision is not a minor implementation detail - it's the structural change that made foundation models possible in the first place. Classical ML's reliance on explicit, human-provided labels created a hard ceiling on how much data a team could realistically gather, a ceiling that made tasks like medical image labeling prohibitively expensive at scale. Self-supervised learning removed that ceiling for language modeling by turning raw, already-abundant text into its own source of training signal.

For engineering teams, the practical lesson isn't that labeling has become obsolete - it's that labeling effort should now be deployed more surgically. Classical supervised pipelines still need comprehensive, well-specified labeled datasets when a narrow task demands one. Foundation-model-powered applications instead benefit most from small, high-quality labeled sets used for fine-tuning, instruction tuning, and evaluation, layered on top of a model that already absorbed broad competence from a self-supervised pretraining process at a scale no manually labeled dataset could match.

References

1. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. "ImageNet Classification with Deep Convolutional Neural Networks." NeurIPS, 2012. (The AlexNet paper)
2. Deng, Jia et al. "ImageNet: A Large-Scale Hierarchical Image Database." CVPR, 2009.
3. Radford, Alec et al. "Improving Language Understanding by Generative Pre-Training." OpenAI, 2018. (GPT and autoregressive language modeling)
4. Ouyang, Long et al. "Training Language Models to Follow Instructions with Human Feedback." NeurIPS, 2022. (InstructGPT / RLHF)
5. Huyen, Chip. AI Engineering: Building Applications with Foundation Models. O'Reilly Media, 2024.
6. Huyen, Chip. Designing Machine Learning Systems. O'Reilly Media, 2022.
7. Vaswani, Ashish et al. "Attention Is All You Need." NeurIPS, 2017.
8. scikit-learn Developers. "Supervised Learning." scikit-learn.org documentation.