paulserban.eu

Writing Edition

Paul Serban

Writing, snippets & book notes

Vol. 2026

June 09, 2024

Prometheus vs. Mimir: What They Are, Why They're Different, and How to Choose

A practical guide to scaling metrics observability beyond a single Prometheus server

Introduction

If you run any modern infrastructure, you have almost certainly run into Prometheus. It has been the default metrics collection and alerting system for cloud-native environments since its graduation from the Cloud Native Computing Foundation (CNCF) in 2018, and it ships as the metrics backbone inside Kubernetes distributions, service meshes, and countless open-source exporters. What is less obvious to teams adopting it is that Prometheus was designed as a single-node system, and that design choice - deliberate, and mostly a good one - eventually collides with the needs of larger organizations that want long-term retention, high availability, and a single pane of glass across dozens or hundreds of clusters.

Grafana Mimir enters the picture at exactly that collision point. It is not a replacement for Prometheus in the way that, say, VictoriaMetrics or Thanos might be framed as "Prometheus alternatives". Instead, Mimir is best understood as a horizontally scalable, multi-tenant storage and query backend that speaks the Prometheus remote-write and PromQL protocols. This article walks through what each system actually does, why the split between "collection" and "long-term storage" exists at all, how to reason about the trade-offs, and a concrete framework for deciding when you need to make the jump from plain Prometheus to something like Mimir.

Context: Why Metrics Systems Split Into Two Layers

Prometheus was built around a pull-based model: a central Prometheus server scrapes HTTP endpoints on a schedule, parses the exposed metrics, and writes them to a local time-series database (TSDB) on disk. This is elegant for a single team running a single cluster. Service discovery, scraping, storage, the PromQL query engine, and alerting all live in one binary, which makes Prometheus trivial to reason about and operationally simple to run. The CNCF's own maturity assessment and the original 2015 SoundCloud origin story both emphasize this simplicity as a deliberate design goal, not an accidental limitation.

The trouble starts at scale. A single Prometheus server's local TSDB is bound to the disk of the machine it runs on, which caps retention (commonly 15-90 days in practice) and makes high availability awkward - you either run two independent Prometheus replicas that scrape the same targets (doubling load and requiring downstream deduplication) or you accept a single point of failure. Neither option gives you a global query view across regions, clusters, or business units, and neither gives you cheap, durable, multi-year retention, because local disk is neither cheap nor durable at that timescale.

This is the gap that "remote storage for Prometheus" systems fill. The pattern is consistent across the ecosystem: Cortex (the CNCF project Mimir forked from), Thanos, and VictoriaMetrics's cluster mode all separate the concerns of scraping (still done by Prometheus, or a Prometheus-compatible agent) from long-term storage and global querying (handled by a separate, horizontally scalable system). Grafana Labs open-sourced Mimir in 2022, positioning it explicitly as the evolution of their earlier contributions to Cortex, with a stated design goal of supporting extremely high cardinality and scale - Grafana Labs has published benchmarks and case studies describing production Mimir clusters ingesting tens of millions of active series, which is roughly the scale at which a single Prometheus instance becomes impractical regardless of hardware.

Deep Technical Explanation: How Prometheus and Mimir Actually Work

Prometheus's internal architecture is a monolith by design. The scrape loop discovers targets via one of its service discovery integrations (Kubernetes SD, Consul, EC2, file-based SD, and others), pulls metrics over HTTP in the OpenMetrics or legacy Prometheus text exposition format, and appends samples to an in-process TSDB. That TSDB uses a write-ahead log (WAL) for crash recovery and organizes data into immutable, time-bounded blocks on disk, each with its own index for label-based lookups. PromQL, Prometheus's query language, operates directly against this local storage. Alerting rules and recording rules are evaluated by the same process on a fixed interval. Everything - ingestion, storage, querying, rule evaluation - shares one address space and one failure domain.

Mimir decomposes that monolith into independently scalable microservices, a pattern it inherited from Cortex, which itself was inspired by scalable database architectures like Google's Monarch and, more directly, by the "big table"-style separation of write and read paths seen in systems like Apache Druid. The distributor is the entry point: it receives remote-write requests (Prometheus's standard protocol for pushing samples to an external system), validates them, and shards series across ingesters using consistent hashing. Ingesters hold recent data in memory and periodically flush immutable blocks to object storage (Amazon S3, Google Cloud Storage, or Azure Blob Storage). The compactor merges and deduplicates those blocks in the background to keep query performance predictable as data accumulates. On the read path, queriers accept PromQL queries, fan them out to ingesters for recent data and to store-gateways for historical data held in object storage, and merge the results.

The practical upshot of this split is that each component scales independently and horizontally. If your bottleneck is ingest volume, you add distributors and ingesters. If your bottleneck is query fan-out across a year of history, you scale store-gateways and queriers. None of this is available to a single Prometheus process, whose scrape loop, storage engine, and query engine all compete for the same CPU and memory on one machine. Mimir also adds native multi-tenancy - every request carries a tenant ID, and Mimir enforces per-tenant limits and isolation - which is essential for platform teams running observability as a shared service across many internal teams, something vanilla Prometheus has no concept of.

It's worth being precise about what stays the same across both systems, because this is where a lot of confusion comes from. Mimir does not replace PromQL; it implements it, aiming for drop-in compatibility so that existing dashboards, alerting rules, and client tooling built against Prometheus's query API continue to work unchanged. It does not replace the scrape loop either - most Mimir deployments still run Prometheus (or Grafana's lighter-weight Grafana Agent / OpenTelemetry Collector) at the edge to do service discovery and scraping, then configure remote_write to ship samples into Mimir. Mimir's job starts where Prometheus's local storage would otherwise take over.

Implementation: What This Looks Like in Practice

Getting Prometheus to ship data into Mimir is a configuration change, not a rewrite. You keep your existing scrape configuration and service discovery exactly as it is, and add a remote_write block pointing at Mimir's distributor endpoint. The example below shows a realistic configuration including basic authentication and a sensible queue configuration for reliability under load:

# prometheus.yml
global:
  scrape_interval: 30s
  evaluation_interval: 30s

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true

remote_write:
  - url: https://mimir.internal.example.com/api/v1/push
    basic_auth:
      username: "team-payments"
      password_file: /etc/prometheus/secrets/mimir_token
    headers:
      X-Scope-OrgID: team-payments
    queue_config:
      capacity: 10000
      max_shards: 50
      max_samples_per_send: 2000
      batch_send_deadline: 5s
    metadata_config:
      send: true

The X-Scope-OrgID header is doing real work here: it is how Mimir identifies which tenant a given write belongs to, enforcing isolation and per-tenant rate limits without requiring separate Mimir clusters per team. On the query side, because Mimir is PromQL-compatible, teams typically point Grafana at Mimir's query-frontend endpoint exactly as they would point it at a Prometheus server, and existing dashboards work unmodified. It's common, though, to validate that assumption programmatically before migrating dashboards wholesale, since subtle differences in default step alignment or lookback behavior can produce slightly different results at query boundaries. A small Python script using the standard PromQL HTTP API is enough to diff results between the two backends during a migration:

import requests
from datetime import datetime, timedelta

def query_range(base_url, query, start, end, step="60s", org_id=None):
    headers = {"X-Scope-OrgID": org_id} if org_id else {}
    resp = requests.get(
        f"{base_url}/api/v1/query_range",
        params={
            "query": query,
            "start": start.isoformat("T") + "Z",
            "end": end.isoformat("T") + "Z",
            "step": step,
        },
        headers=headers,
        timeout=30,
    )
    resp.raise_for_status()
    return resp.json()["data"]["result"]

def compare_backends(query: str, minutes: int = 30):
    end = datetime.utcnow()
    start = end - timedelta(minutes=minutes)

    prom_result = query_range("http://prometheus:9090", query, start, end)
    mimir_result = query_range(
        "https://mimir.internal.example.com/prometheus",
        query, start, end, org_id="team-payments",
    )

    prom_series = {tuple(sorted(r["metric"].items())) for r in prom_result}
    mimir_series = {tuple(sorted(r["metric"].items())) for r in mimir_result}

    only_in_prom = prom_series - mimir_series
    only_in_mimir = mimir_series - prom_series

    if only_in_prom or only_in_mimir:
        print(f"Mismatch for query: {query}")
        print(f"  Only in Prometheus: {only_in_prom}")
        print(f"  Only in Mimir:      {only_in_mimir}")
    else:
        print(f"OK: {query} - {len(prom_series)} matching series")

if __name__ == "__main__":
    compare_backends('rate(http_requests_total{job="checkout"}[5m])')

Trade-offs and Pitfalls

The most common mistake teams make is treating "add Mimir" as a purely additive decision with no cost. It is not. Running Mimir well means operating a distributed system: you now have distributors, ingesters, compactors, store-gateways, and a query-frontend, each of which needs its own capacity planning, monitoring, and upgrade strategy, plus an object storage bucket and typically a key-value store (memberlist or Consul-based) for ring coordination between components. Grafana Labs ships a "monolithic" deployment mode specifically to soften this, letting you run all of Mimir's components in one process for smaller deployments, but that mode still carries more operational surface area than a single Prometheus binary, and it is explicitly described in Grafana's own documentation as a stepping stone toward the fully distributed "microservices" mode rather than a permanent end state for large environments.

Cardinality is the second trap, and it bites both systems, just differently. Prometheus's local TSDB will visibly degrade - high memory usage, slow queries, eventual OOM kills - when label cardinality explodes (for example, putting a user ID or a raw request path into a label). Mimir's distributed architecture tolerates much higher aggregate cardinality, but it does not make the underlying problem free; high-cardinality series still consume ingester memory and store-gateway index space, and Mimir enforces configurable per-tenant series limits precisely because unbounded cardinality can still degrade a shared cluster for every tenant on it, not just the offending one. Teams migrating to Mimir sometimes read "handles millions of series" as license to stop being disciplined about label design, and then are surprised when they hit tenant limits or see costs climb.

The third trade-off is latency and consistency semantics. Because Mimir's write path involves a distributor hashing and forwarding samples to ingesters, and its read path involves querying both in-memory ingester data and object-storage blocks, there is inherently more moving parts between "a sample was scraped" and "that sample is queryable" compared to Prometheus's tight local loop. For most dashboarding and alerting use cases this added latency (typically low seconds) is invisible, but for latency-sensitive automated remediation systems that query metrics as a control-loop input, it's a real design constraint worth testing under load rather than assuming away.

Best Practices: A Decision Framework

Start from the actual constraint you are hitting rather than the tool you have heard about. If a single team runs a single cluster, needs 30-90 days of retention, and already has reasonable label hygiene, plain Prometheus with local storage and a well-configured Alertmanager is not a legacy choice you should feel pressure to outgrow - it is the correct, boring, low-operational-cost answer, and the CNCF's own graduation of Prometheus reflects how well this model works at that scale. The signal to look elsewhere is specific and usually one of: you need retention measured in months or years for compliance or trend analysis, and local disk cannot economically hold it; you need a global query view across many clusters or regions that federation (Prometheus's built-in but limited cross-server query mechanism) cannot satisfy well; you are running observability as a platform for multiple internal teams and need real multi-tenant isolation and per-team rate limiting; or you need high availability without doubling scrape load and hand-rolling deduplication.

When those signals are present, the next decision is not just "Mimir versus nothing" but "which remote-storage system fits our operational appetite". Mimir is a strong default if you are already standardizing on the Grafana stack (Grafana, Loki, Tempo) and want one vendor's opinionated, well-documented path, and if you have or are willing to build the platform-engineering capacity to run a distributed system - or you use Grafana Cloud's managed offering and sidestep that operational cost entirely. Thanos is worth evaluating if you want to keep each Prometheus server's local TSDB as the source of truth and layer a global query and long-term storage view on top without fully centralizing ingestion. VictoriaMetrics's cluster mode is worth evaluating if resource efficiency and simpler operations than a full Mimir deployment matter more to you than deep Grafana-ecosystem integration. In every case, the honest first step is measuring your actual active series count and desired retention, because that number, more than any architectural preference, tells you whether you are solving a real problem or borrowing complexity you don't need yet.

Once you do adopt a scalable backend, invest early in cardinality governance rather than treating it as a later cleanup task: enforce label conventions in CI (reject metrics with unbounded label values like raw user IDs or full URLs), set per-tenant series limits deliberately rather than leaving defaults, and build a dashboard of your own ingestion cardinality over time so growth is visible before it becomes an incident. This is cheaper to do before migration than after, because bad habits that were merely annoying on a single Prometheus server become expensive on a shared multi-tenant platform where one team's mistake can degrade the experience for everyone else on the cluster.

Finally, treat the migration itself as reversible and incremental. Because Mimir speaks Prometheus's remote-write protocol and PromQL, you can run Prometheus with remote-write enabled while continuing to serve queries from local storage during a transition period, validate query parity (as in the comparison script above), and only cut dashboards and alerting rules over to the new backend once you have confidence. There is no requirement to migrate all teams or all clusters simultaneously, and a phased rollout by team or by environment is both lower-risk and easier to roll back than a single cutover.

Analogies and Mental Models

A useful mental model is the difference between a personal notebook and a shared, searchable archive. Prometheus's local TSDB is the notebook: fast to write in, fast to flip through, entirely under your control, but it lives on your desk, has a finite number of pages, and nobody else can search it while you're using it. It's the right tool for one person's (or one team's) day-to-day work. Mimir is closer to a shared document management system with proper indexing: multiple people can write to it concurrently without stepping on each other, it scales to far more pages than any one desk could hold, and it supports searching across everyone's contributions at once - but you now need a system administrator, storage budget, and access-control policy, none of which the notebook ever required.

Another way to frame the relationship is as a factory floor versus a warehouse. Prometheus's scrape loop and short-term storage are the factory floor: high-throughput, tightly coupled, optimized for fast local access to recent work-in-progress. Mimir's ingesters, compactors, and object-storage-backed blocks are the warehouse: optimized for durable, cheap, long-term storage of finished goods, accessed less frequently per item but at much larger scale. You don't put your factory floor in the warehouse, and you don't try to run factory-floor operations out of warehouse infrastructure - you build a conveyor belt (remote_write) between the two and let each do what it's good at.

Key Takeaways

Conclusion

Prometheus and Mimir are not competitors; they are two layers of the same stack solving different problems. Prometheus excels at the hard, latency-sensitive work of discovering targets, scraping them reliably, and evaluating alerts quickly against recent data, all within an operationally simple, single-binary footprint that has earned its status as the default choice for cloud-native metrics. Mimir exists because that same simplicity becomes a constraint once an organization needs retention, scale, or multi-tenancy that a single machine's disk and a single process's query engine cannot provide, and it solves that specific problem by decomposing storage and querying into a horizontally scalable, object-storage-backed system that still speaks Prometheus's native protocols.

The decision, then, is rarely "Prometheus or Mimir". It's "how much of my metrics stack still fits comfortably on one Prometheus server, and at what point does that stop being true". Answering that honestly - with real numbers on series count, retention needs, and team structure rather than assumptions about what "enterprise-grade" observability requires - will tell you far more than any comparison chart. For many teams, a well-tuned Prometheus deployment with sensible retention and federation is genuinely sufficient for years. For others, particularly platform teams serving many internal customers or organizations with real compliance-driven retention requirements, Mimir (or a comparable remote-storage system) is the natural next step, not because Prometheus failed, but because it was never designed to solve that particular problem in the first place.

References