paulserban.eu

Writing Edition

Paul Serban

Writing, snippets & book notes

Vol. 2026

August 07, 2023

Cardinality in Observability: What It Is, Why It Breaks Your Monitoring, and How to Control It

A practical, engineering-first guide to metric, trace, and log cardinality - from first principles to production-grade mitigation strategies

Introduction

Every engineer who has run a production system past a certain scale has a cardinality story. It usually starts innocently: someone adds a user_id label to a metric to make debugging easier, or a request_id tag to a trace span "just in case." Weeks later, the metrics backend is falling over, dashboards time out, and the monthly observability bill has tripled. Nobody changed the traffic pattern. What changed was cardinality - the number of unique combinations of label values a monitoring system has to track - and it is one of the most consequential, least understood variables in observability engineering.

This article treats cardinality as a first-class engineering concern rather than an operational footnote. We will define it precisely, explain the mechanics of why it causes systems to degrade, walk through concrete code examples of how cardinality is created and controlled in real instrumentation, and then move into the trade-offs, pitfalls, and best practices that separate teams who scale their observability stack gracefully from teams who rebuild it every eighteen months. The goal is not just to help you avoid an outage caused by an exploding label set - it's to help you build a mental model that lets you reason about cardinality the way you already reason about time complexity or memory allocation.

What Cardinality Actually Means in Observability

In the context of metrics, cardinality refers to the number of unique time series produced by a metric name combined with all possible combinations of its label (or dimension) values. A metric like http_requests_total with a single label method that can be GET, POST, PUT, or DELETE has a cardinality of four - four unique time series. Add a status_code label with ten possible values, and cardinality multiplies to forty. Add a user_id label with a million possible values, and cardinality explodes to forty million. This multiplicative behavior is the crux of the problem: cardinality does not add across dimensions, it multiplies.

This matters because most metrics backends - Prometheus, VictoriaMetrics, InfluxDB, and cloud-managed equivalents like Amazon Managed Service for Prometheus or Google Cloud Monitoring - store data as time series, where each unique combination of metric name and label set is allocated its own storage stream, its own index entry, and its own memory-resident state for active scraping or ingestion. The system does not know in advance how many series will exist; it discovers them as data arrives. A well-behaved metric with bounded, low-cardinality labels behaves predictably. A metric with an unbounded or high-cardinality label - anything derived from user input, IDs, timestamps, or free-text fields - can grow without limit, and the backend has no way to reject or throttle it gracefully at the point of ingestion in most default configurations.

Cardinality is not unique to metrics. It applies to structured logs indexed by field, and to distributed traces where span attributes and tags create similarly unbounded index growth in trace storage backends. The mechanics differ by system, but the underlying principle - unique combinations of dimensions create unique storage and query overhead - is the same across the three pillars of observability.

Why High Cardinality Breaks Systems

The most direct failure mode is memory pressure. Prometheus, for example, keeps the "head block" of recently ingested data in memory before it is compacted to disk, and it maintains an inverted index mapping label names and values to the time series that contain them. This index is what makes queries like sum(rate(http_requests_total{status_code="500"}[5m])) fast - the system doesn't have to scan every series, just the ones matching the label filter. But the index itself grows with cardinality, and at high enough cardinality, the index and head block consume enough memory that the process is OOM-killed or the host runs out of resources entirely. This is well documented in Prometheus's own operational guidance, which explicitly warns against labels with unbounded value sets.

The second failure mode is query performance. Aggregation queries in PromQL, LogQL, or similar query languages have to touch every series matching a selector before they can reduce them with functions like sum, avg, or histogram_quantile. A query that seemed instantaneous at ten thousand series can take tens of seconds at ten million series, and dashboards built on Grafana or similar tools will time out or degrade the entire team's ability to triage incidents - precisely when fast queries matter most.

The third, and often most painful for organizations, is cost. Cloud observability vendors - Datadog, New Relic, Honeycomb, Grafana Cloud, and others - commonly price custom metrics based on the number of unique time series or unique tag combinations ingested per month. A single instrumentation change that adds a high-cardinality label can silently multiply a team's monthly bill by an order of magnitude, and because billing granularity is often opaque until the invoice arrives, this frequently surfaces as a finance problem before it surfaces as an engineering problem. Several vendors, including Datadog and Honeycomb, publish guidance specifically warning customers about cardinality-driven cost and performance impact, which reflects how common this failure pattern is across the industry.

Beyond memory, query latency, and cost, there is a subtler failure: alerting reliability. Alerting rules that aggregate over high-cardinality series can silently stop firing or fire on stale data if the underlying query times out or is throttled, meaning cardinality problems can directly translate into missed incidents.

Deep Technical Explanation: The Mechanics of Cardinality Explosion

To reason precisely about cardinality, it helps to think of a metric as a function from a label schema to a set of time series. If a metric has labels L1, L2, ..., Ln with cardinalities |L1|, |L2|, ..., |Ln| respectively, the theoretical maximum number of time series is the product |L1| x |L2| x ... x |Ln|. In practice, actual cardinality is usually lower than this theoretical maximum because not all combinations occur (a mobile client never sends a desktop_browser value, for instance), but the growth remains combinatorial, not additive, and any label with a naturally large or unbounded domain - user IDs, session IDs, IP addresses, full URLs with query strings, container instance IDs in an autoscaling environment - dominates the total.

This is why the single most dangerous cardinality mistake is attaching a high-cardinality field directly as a metric label instead of as contextual data elsewhere. Traces and logs are built to carry high-cardinality context precisely because they are stored and queried differently: a trace span can hold a user_id attribute because a single trace is retrieved by ID or through sampling, not aggregated across the entire attribute space the way a metric is. Confusing "data that belongs in a metric label" with "data that belongs in an event attribute" is arguably the single most common instrumentation error observability teams make. This distinction is central to the design philosophy behind OpenTelemetry and to the wide-events model advocated by observability practitioners such as Charity Majors and the Honeycomb team, who argue that rich, high-cardinality context belongs in structured events, while metrics should remain a small, bounded, pre-aggregated summary layer on top.

It's also worth understanding cardinality churn - a related but distinct problem. Churn refers to how quickly the set of active series changes over time, which happens heavily in environments using ephemeral infrastructure. If every container or pod gets its own instance label and pods are recycled every few minutes by an autoscaler, the metrics backend is constantly allocating new series and expiring old ones. Even if the cardinality at any single point in time is bounded, high churn still stresses the index and can cause similar operational symptoms to raw cardinality explosion, particularly in Prometheus's TSDB, which was originally optimized for relatively stable, long-lived series.

Kubernetes environments are a canonical case study here, since dynamic pod names, revision hashes, and node labels are frequently, and mistakenly, exposed as Prometheus labels by default exporters or misconfigured relabeling rules, compounding both cardinality and churn simultaneously.

Implementation: Practical Examples of Controlling Cardinality

The cleanest way to prevent cardinality problems is to design instrumentation with cardinality budgets in mind from the start. Below is a Python example using the prometheus_client library that demonstrates the wrong and right way to instrument an HTTP handler.

from prometheus_client import Counter, Histogram
from flask import Flask, request

app = Flask(__name__)

# ANTI-PATTERN: user_id and full path create unbounded cardinality.
# Do not do this in production.
bad_requests_total = Counter(
    "http_requests_total_bad",
    "Total HTTP requests (anti-pattern)",
    ["user_id", "path"],
)

# BETTER: bounded labels only. Route templates instead of raw paths,
# and no per-user dimension on the metric itself.
requests_total = Counter(
    "http_requests_total",
    "Total HTTP requests",
    ["method", "route", "status_code"],
)

request_latency_seconds = Histogram(
    "http_request_duration_seconds",
    "HTTP request latency",
    ["method", "route"],
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10),
)


@app.route("/orders/<order_id>")
def get_order(order_id: str):
    # route is the templated path, not the raw request.path,
    # which would include order_id and blow up cardinality.
    route = "/orders/:order_id"
    with request_latency_seconds.labels(method="GET", route=route).time():
        result = fetch_order(order_id)
    requests_total.labels(method="GET", route=route, status_code="200").inc()
    return result

The key discipline here is using the route template (/orders/:order_id) rather than the interpolated path (/orders/8471) as a label value. This single pattern - normalizing dynamic path segments before they touch a metric label - eliminates the majority of accidental cardinality explosions seen in HTTP instrumentation across every language and framework.

The same discipline applies in OpenTelemetry-instrumented TypeScript services, where the temptation is to attach request-scoped identifiers to metric attributes instead of span attributes. The example below shows the OpenTelemetry JS SDK used correctly: high-cardinality identifiers go on the span (a trace-level construct queried per-request), while only bounded dimensions go on the counter (a metric-level construct that is aggregated).

import { trace, metrics } from "@opentelemetry/api";

const tracer = trace.getTracer("order-service");
const meter = metrics.getMeter("order-service");

const requestCounter = meter.createCounter("http_requests_total", {
  description: "Count of HTTP requests by method, route, and status",
});

async function handleGetOrder(orderId: string, method: string) {
  const span = tracer.startSpan("get_order");
  // High-cardinality context belongs on the span, not the metric.
  span.setAttribute("order.id", orderId);
  span.setAttribute("http.method", method);

  try {
    const result = await fetchOrder(orderId);
    span.setAttribute("http.status_code", 200);

    // Only bounded, low-cardinality attributes go on the metric.
    requestCounter.add(1, {
      "http.method": method,
      "http.route": "/orders/:order_id",
      "http.status_code": "200",
    });

    return result;
  } catch (err) {
    span.recordException(err as Error);
    requestCounter.add(1, {
      "http.method": method,
      "http.route": "/orders/:order_id",
      "http.status_code": "500",
    });
    throw err;
  } finally {
    span.end();
  }
}

For teams that inherit systems already emitting high-cardinality metrics, relabeling at the collection layer is the most common remediation. Prometheus supports metric_relabel_configs to drop or rewrite labels before ingestion, and the OpenTelemetry Collector offers equivalent processors, such as the attributes and transform processors, to strip or hash high-cardinality fields before they reach a backend.

# prometheus.yml - drop a known high-cardinality label before storage
scrape_configs:
  - job_name: "order-service"
    metric_relabel_configs:
      - action: labeldrop
        regex: "user_id|session_id|request_id"

This kind of relabeling is a stopgap, not a fix - it addresses the symptom at the collection layer while the underlying instrumentation still emits the problematic labels. It is useful for triage during an incident, but the durable solution is always to correct instrumentation at the source.

Trade-offs and Common Pitfalls

Cardinality control is not free, and treating it as a pure cost-reduction exercise misses the real trade-off: cardinality is also what gives observability data its diagnostic power. A metric with zero labels tells you a single global number. A metric with a region and status_code label lets you isolate a regional outage. The entire value proposition of dimensional metrics is that you can slice by the dimensions that matter - and cutting cardinality too aggressively can leave you with dashboards that show "errors went up" without any way to determine where or why. The engineering skill here is not minimizing cardinality, it's spending cardinality budget deliberately on dimensions with genuine diagnostic value, and pushing anything higher-cardinality down into logs or traces, where it can still be queried, just through a different access pattern.

A frequent pitfall is discovering cardinality problems only after they've already caused an incident, because most teams do not monitor the cardinality of their own metrics. Prometheus exposes prometheus_tsdb_head_series and related internal metrics precisely so operators can track total active series over time, yet many teams never build a dashboard or alert on this signal until after their first cardinality-driven outage. The same blind spot exists on the tracing side: teams rarely track the cardinality of span attributes until a trace backend's ingestion costs or query latency forces the question.

A second pitfall is conflating high cardinality with high dimensionality. Dimensionality is the number of distinct label keys on a metric; cardinality is the number of unique value combinations across those keys. A metric can have low dimensionality (three labels) and still have catastrophic cardinality if just one of those three labels has an unbounded value domain. Teams sometimes respond to a cardinality incident by removing labels indiscriminately, which reduces both dimensionality and diagnostic value without necessarily targeting the actual offending label.

A third pitfall, particularly relevant to organizations adopting OpenTelemetry, is assuming that traces are immune to cardinality problems because they are not aggregated the way metrics are. This is only partially true. Trace backends still build indexes over span attributes to support search and filtering, and an attribute with extremely high cardinality - like a raw SQL query string used as a span attribute - can still degrade index performance and searchability even though it doesn't cause the same kind of aggregation-time blowup that metrics experience. Sampling strategies, discussed further below, are the primary lever for managing trace-side volume and cost, but they don't eliminate attribute-cardinality concerns for the traces that are retained.

Best Practices for Managing Cardinality at Scale

The most effective long-term strategy is establishing cardinality ownership as part of the code review process, not as a reactive operations task. This means treating a new metric label the way you'd treat a new database index: something that has a cost, needs a justification, and should be reviewed before it ships. Some organizations formalize this with lint rules or CI checks that flag known high-cardinality field names (user_id, email, ip_address, request_id) when they appear in metric-emitting code paths, catching the mistake before it reaches production rather than after a bill spike.

Choosing the right telemetry signal for the right data is the second pillar of good cardinality hygiene. As a working rule: metrics should carry dimensions with a small, enumerable set of values known in advance (HTTP method, status code class, environment, region, service name); logs and trace attributes should carry the high-cardinality, per-request context (user IDs, order IDs, trace IDs, raw error messages). This is not a stylistic preference - it reflects how each backend is architected to store and query its data, and instrumenting against the grain of that architecture is what produces cardinality incidents in the first place.

Sampling is the primary lever for controlling volume and cardinality-adjacent cost on the tracing side. Head-based sampling, where a decision to keep or drop a trace is made at the start of the request, is simple and cheap but risks discarding the rare, interesting traces - the slow outlier, the one that errors - that engineers actually want to investigate. Tail-based sampling, where the decision is made after the full trace completes, can preferentially retain error traces and high-latency traces while dropping routine ones, but it requires buffering complete traces in the collector, which has its own memory cost. The OpenTelemetry Collector supports both approaches through its probabilistic_sampler and tail_sampling processors, and most mature tracing setups use tail-based sampling for exactly this reason: it preserves diagnostic value while controlling ingestion volume.

For metrics that genuinely need higher-cardinality slicing - the classic example being per-customer SLA tracking in a B2B SaaS product - a common pattern is to maintain a small number of "gold path" high-cardinality metrics for the customers or endpoints that matter most, while keeping the general instrumentation bounded. Some organizations achieve this with recording rules that pre-aggregate raw high-cardinality data into a smaller set of derived series, trading some storage and compute at ingestion time for dramatically cheaper query-time performance - a classic write-time versus read-time trade-off that should feel familiar to anyone who has designed a database schema.

Finally, cardinality limits should be enforced, not just recommended. Prometheus supports a sample_limit per scrape target to reject scrapes that exceed a threshold, functioning as a circuit breaker against a single misbehaving service taking down a shared metrics backend. Cloud-managed metrics platforms typically offer similar quota or cardinality-limiting controls at the account or namespace level. Treating these limits as a safety net - not a substitute for good instrumentation - gives teams room to catch mistakes before they become outages.

Advanced Considerations: Cardinality Across the Observability Stack

As organizations mature past basic dashboards, cardinality management starts to intersect with more advanced architectural decisions. One is the use of exemplars - a feature supported by Prometheus's native histogram format and OpenMetrics - which attach a single representative trace ID to a metric bucket without requiring the trace ID itself to become a metric label. This gives engineers a "click-through" path from an aggregated metric spike directly into an individual trace, effectively getting the diagnostic benefit of high-cardinality data without paying its storage cost, since only one exemplar is retained per bucket rather than one series per unique value.

Another advanced pattern involves cardinality-aware storage engines. Systems like VictoriaMetrics and Thanos (built on top of or alongside Prometheus's storage model) are specifically engineered to handle larger cardinality more gracefully than vanilla Prometheus, through techniques like better compression of label sets and more efficient inverted indexing. Choosing a storage backend informed by expected cardinality growth, rather than retrofitting one after an incident, is a decision that increasingly belongs in initial platform architecture discussions rather than being deferred to whichever team hits the wall first.

A less obvious but increasingly important area is cardinality in the context of cost attribution and FinOps for observability itself. As organizations centralize telemetry through a shared collector layer - often the OpenTelemetry Collector - that layer becomes a natural point to apply per-team or per-service cardinality budgets, similar to how cloud cost allocation tags work for infrastructure spend. Some organizations implement this using the Collector's transform processor combined with custom logic to reject or downsample telemetry that exceeds a team's allocated series budget, turning cardinality from an invisible shared-tragedy-of-the-commons problem into something individual teams can see and are accountable for.

Analogies and Mental Models

The clearest mental model for cardinality is a spreadsheet. Imagine a metric as a spreadsheet where each label is a column and each unique combination of column values is a row. A metric with no labels is a spreadsheet with one row - a single running total. Add a region column with five possible values, and you now have five rows. Add a user_id column with a million possible values, and the spreadsheet needs a million rows for every combination of region and user that actually occurs. The backend has to allocate storage, memory, and index space for every row that exists, whether or not you ever query it - which is precisely why an unbounded column turns a five-row spreadsheet into one with millions of rows almost overnight.

A second useful analogy is a phone book versus a diary. A metric behaves like a phone book: it's organized for fast lookup across a bounded, structured set of entries (names map to numbers, methods map to counts), and it works beautifully as long as the number of distinct entries stays manageable. A trace or log event behaves like a diary entry: rich, specific, full of high-cardinality detail about one particular moment, but not something you'd try to aggregate across a million diary pages to compute a single number. Cardinality problems occur when engineers try to make the phone book behave like a diary - cramming unbounded, per-entry detail into a data structure designed for bounded, structured lookup.

Key Takeaways

The 80/20 Insight

The overwhelming majority of cardinality incidents trace back to a single root cause: an identifier that should have lived in a log or trace was instead attached as a metric label. If you fix nothing else about your observability practice, enforcing the rule "no unbounded values as metric labels, ever" during code review will eliminate the large majority of cardinality-driven outages and cost overruns that teams actually experience in production. Everything else in this article - sampling strategies, recording rules, cardinality-aware storage engines, exemplars - matters at scale, but it is optimization on top of a foundation that this one rule protects.

Conclusion

Cardinality is not a niche operational detail; it is a structural property of your telemetry data that determines whether your observability stack scales gracefully or becomes a recurring source of outages and unexpected cost. The underlying principle is simple even though its consequences are not: metrics are built for bounded, aggregable dimensions, while logs and traces are built to carry the rich, high-cardinality context that metrics cannot afford. Once that distinction is internalized, most cardinality mistakes become obvious in code review rather than surprising in a postmortem.

The teams that manage this well are not the ones with the fewest labels - they're the ones who spend their cardinality budget deliberately, monitor it as a first-class signal, and treat instrumentation changes with the same scrutiny they'd apply to a schema migration. As distributed systems continue to grow in scale and ephemerality, particularly with Kubernetes and serverless architectures generating naturally high-churn identifiers, cardinality discipline will only become more central to reliable, cost-effective observability - not less.

References

A quick flashcard preview - one example of each question type in this post. This is a demo of what you'll find in the full quiz app.

Flashcard preview - 1 / 8Multiple Choice

multiple choice - advanced - auto-graded

Why does Prometheus's inverted index create a trade-off as cardinality increases?

Choose an answer