paulserban.eu

Writing Edition

Paul Serban

Writing, snippets & book notes

Vol. 2026

August 08, 2023

Grafana Loki Explained: What It Is, Why It Matters, and How to Run It in Production

A practical guide to log aggregation with Loki - from architecture and LogQL to scaling patterns and how it compares to Elasticsearch, Splunk, and OpenSearch

Introduction

Every engineering team eventually hits the same wall: logs are scattered across dozens of services, containers restart and take their stdout with them, and the only way to debug a production incident is to SSH into a box that may no longer exist. Log aggregation systems exist to solve exactly this problem, and for the last several years Grafana Loki has become one of the default answers, particularly for teams already standardized on Prometheus and Grafana for metrics.

Loki is often described as "Prometheus, but for logs," and that shorthand is useful but incomplete. It captures the philosophy - label-based indexing instead of full-text indexing - without explaining the trade-offs that philosophy implies. This article walks through what Loki actually is, the architectural decisions that make it cheap to run at scale, how to configure and query it in practice, where it falls short, and how it compares to the other major players in the observability logging space: the Elastic Stack, OpenSearch, Splunk, and Datadog Logs.

The Problem Loki Was Built to Solve

Before Loki, the dominant open-source pattern for log aggregation was the "ELK stack" - Elasticsearch, Logstash, and Kibana. Elasticsearch indexes every field of every log line using an inverted index, which makes it extraordinarily flexible for ad hoc search but also expensive. Full-text indexing means storage costs scale with the cardinality and volume of log content itself, not just its size on disk. Teams ingesting terabytes of logs per day routinely found that their Elasticsearch clusters cost more to run than the applications generating the logs.

Grafana Labs (then Grafana Labs, formed from the team behind Grafana and Cortex) released Loki in 2018 with a different premise: most of the time, engineers don't search logs by arbitrary free-text queries across the entire fleet. They search by known dimensions - service name, namespace, pod, environment, log level - and then grep through the matching lines for a specific string or pattern. If that's true, you don't need to index every word of every log line. You only need to index the metadata (labels), and you can store the actual log content as compressed, unindexed blobs that get scanned only when a query actually needs them.

This single design decision cascades into almost everything else about Loki. Because the index is small (just label sets, not content), it can live in a lightweight key-value store or even embedded storage like BoltDB or TSDB files. Because the log content is stored as compressed chunks in cheap object storage (S3, GCS, Azure Blob, or on-prem equivalents like MinIO), storage costs drop by an order of magnitude compared to a fully-indexed system for equivalent log volume. The trade-off, which we'll return to later, is that queries which don't specify tight label selectors - the "give me every log line containing this string, anywhere" style query - become expensive full scans instead of fast index lookups.

The other forcing function behind Loki's design was operational consistency. Teams running Prometheus for metrics already understood label-based data models, PromQL, and the Grafana visualization layer. Loki was built to slot into that existing mental model and toolchain rather than requiring a second, unrelated system with its own query language, storage format, and operational quirks. LogQL, Loki's query language, was deliberately designed to look and feel like PromQL, and Grafana added first-class Loki data source support so metrics and logs could sit in the same dashboards, correlated by time and shared labels.

Architecture: How Loki Actually Works

Loki's architecture separates concerns into distinct components, which can either run as a single monolithic binary (-target=all) for small deployments or be split into independently scalable microservices for production-grade, high-volume environments. Understanding these components is essential for both debugging and capacity planning.

Distributors are the entry point for all incoming log data. When an agent like Promtail, Grafana Alloy, Fluent Bit, or the Loki Docker driver pushes logs, the distributor validates the request, checks it against configured limits (rate limits, label cardinality limits, max line size), and hashes the log stream - a stream being a unique combination of label values - to determine which ingester should own it. Distributors are stateless and horizontally scalable, which makes them a natural place to absorb traffic spikes.

Ingesters are the stateful workhorses. Each ingester receives streams from the distributor, batches log lines into per-stream chunks in memory, and periodically flushes those chunks to long-term object storage once they hit a size or time threshold. Ingesters also serve recent, not-yet-flushed data for queries that touch the last few hours. This is why ingester memory and replication factor matter so much operationally: losing an ingester before it flushes means losing unflushed log data, which is why Loki supports a write-ahead log (WAL) to make ingesters crash-recoverable.

Queriers handle LogQL evaluation. A querier receives a query, determines which chunks in object storage (and which in-memory data in ingesters) are relevant based on the label selectors and time range, fetches those chunks, decompresses them, and executes the filter or pattern match. For high query volume, Loki also has a query frontend and query scheduler that split large queries into shards, deduplicate identical concurrent queries, and queue work fairly across tenants - this is what makes Loki viable in multi-team, multi-tenant deployments rather than just single-team setups.

The storage layer itself has evolved. Early Loki used BoltDB shipper for the index and chunk stores like S3 or GCS for content. Modern Loki (v2.8+) defaults to TSDB as the index format, which improves query performance and reduces the operational burden of index compaction. The compactor component runs in the background to merge small chunks, apply retention policies, and deduplicate index entries - without it, long-running Loki clusters accumulate an ever-growing number of small objects that degrade query performance over time.

Implementation: Running and Querying Loki

The fastest way to get a feel for Loki is a local Docker Compose stack with Promtail (or its successor, Grafana Alloy) shipping container logs. The example below sets up Loki, Promtail, and Grafana together.

# docker-compose.yml
version: "3.8"

services:
  loki:
    image: grafana/loki:3.1.0
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki-config.yaml:/etc/loki/local-config.yaml

  promtail:
    image: grafana/promtail:3.1.0
    volumes:
      - /var/log:/var/log
      - /var/run/docker.sock:/var/run/docker.sock
      - ./promtail-config.yaml:/etc/promtail/config.yaml
    command: -config.file=/etc/promtail/config.yaml
    depends_on:
      - loki

  grafana:
    image: grafana/grafana:11.2.0
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    depends_on:
      - loki

A minimal Promtail scrape config demonstrates the label discovery pattern that determines how logs will later be queried:

# promtail-config.yaml
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: [__meta_docker_container_name]
        target_label: container
      - source_labels: [__meta_docker_container_log_stream]
        target_label: stream

Once logs are flowing, LogQL is where the day-to-day value shows up. A LogQL query has two parts: a label matcher (like a PromQL vector selector) and an optional pipeline of filters, parsers, and formatters.

# All error-level logs from the checkout-service in production, last 15 minutes
{namespace="production", app="checkout-service"} |= "error"

# Parse JSON logs and filter on a nested field, then extract latency
{namespace="production", app="checkout-service"}
  | json
  | status_code >= 500
  | line_format "{{.timestamp}} {{.message}} latency={{.latency_ms}}"

# Rate of 5xx errors per service over 5-minute windows, for alerting
sum by (app) (
  rate({namespace="production"} | json | status_code >= 500 [5m])
)

That last query is the pattern that makes Loki genuinely powerful beyond simple grep-in-the-browser use cases: LogQL supports metric queries, turning log streams into time series that can be graphed, alerted on via Grafana Alerting or Loki's own ruler component, and correlated directly with Prometheus metrics on the same dashboard panel.

For teams instrumenting applications directly rather than relying solely on infrastructure-level scraping, pushing structured logs straight to Loki's HTTP API is common in serverless or batch contexts where a sidecar agent isn't practical:

import time
import json
import requests

def push_log(loki_url: str, labels: dict, message: str) -> None:
    timestamp_ns = str(int(time.time() * 1e9))
    payload = {
        "streams": [
            {
                "stream": labels,
                "values": [[timestamp_ns, json.dumps({"message": message})]],
            }
        ]
    }
    response = requests.post(
        f"{loki_url}/loki/api/v1/push",
        json=payload,
        headers={"Content-Type": "application/json"},
        timeout=5,
    )
    response.raise_for_status()

push_log(
    "http://loki:3100",
    {"job": "billing-worker", "env": "production", "level": "error"},
    "Failed to process invoice #48213: downstream timeout",
)

Trade-offs and Common Pitfalls

The label-based indexing model that makes Loki cheap is also its sharpest edge, and teams that migrate from Elasticsearch without understanding this often get burned. High-cardinality labels - anything like user_id, request_id, session_id, or trace_id attached as a label rather than as log content - create an explosion of unique streams. Each unique label combination is a separate stream with its own chunk, and Loki's index grows proportionally with the number of streams, not the volume of data. A well-intentioned engineer who labels logs by user_id to make debugging "easier" can silently degrade an entire cluster's ingestion and query performance, sometimes to the point of ingesters running out of memory. The fix is almost always to keep such identifiers in the log line itself, filterable via LogQL's |=, |~, or | json pipeline stages, rather than as an indexed label.

The second common pitfall is treating LogQL like a full-text search engine. A query without label selectors, or with only very broad ones like {namespace="production"} across a week-long range, forces Loki to fetch and decompress a large number of chunks and scan them line by line. This is fundamentally different from Elasticsearch, where the inverted index makes such searches fast regardless of selectivity. Loki queries are fast when they are narrow in both label scope and time range, and slow when they aren't - which means query patterns that worked fine against a small Elasticsearch cluster can time out against Loki if teams don't adjust their querying habits and dashboard designs accordingly.

Operationally, running Loki in microservices mode introduces real complexity: replication factors, ring membership via Consul or memberlist, WAL configuration, and compactor tuning all require attention that a single-binary deployment doesn't. Grafana Labs' own managed offering (Grafana Cloud) exists partly because self-hosting Loki well, at genuine production scale with multi-tenancy and high availability, is nontrivial work - comparable in kind to the operational investment teams already make in running Prometheus or Thanos at scale.

Best Practices for Production Loki

Design your label schema before you write a single scrape config, not after. A good rule of thumb, echoed in Grafana Labs' own documentation, is to keep the total number of distinct label value combinations for any given job in the low thousands at most, and to reserve labels for dimensions you will actually filter dashboards and alerts by - environment, cluster, namespace, application, and log level are typical candidates. Everything else - request IDs, user IDs, stack traces, arbitrary key-value pairs - belongs in the log line body, extracted at query time with LogQL's json, logfmt, pattern, or regexp parsers.

Set retention and storage tiers deliberately rather than accepting defaults. Because chunks live in object storage, Loki makes it straightforward to apply different retention windows per tenant or even per stream using the compactor's retention configuration, which lets teams keep expensive, high-fidelity logs for a short window (say, 7-14 days for full-text debugging) while retaining derived metrics or sampled logs far longer at negligible cost. This tiering is one of the more underused capabilities of the platform, and it's where a lot of the promised cost savings actually materialize in a bill.

Instrument for structured logging from day one. Loki's parsing pipeline stages - particularly json and logfmt - are dramatically faster and more reliable against structured logs than against unstructured free text parsed with regex. Teams that standardize on structured logging libraries (structlog in Python, pino or winston in Node.js, zerolog or zap in Go) get much more value out of LogQL's metric queries and field extraction, and they avoid the fragile regex patterns that break the moment a log format changes slightly.

Finally, treat Loki alerting and dashboards as first-class citizens alongside Prometheus, not an afterthought. Because LogQL metric queries produce standard Prometheus-compatible time series, Loki's ruler component can evaluate alerting rules directly against log-derived metrics - for example, alerting when the rate of a specific error string crosses a threshold - using the same alerting pipeline (Alertmanager or Grafana Alerting) already in place for infrastructure metrics. This convergence is one of the strongest arguments for adopting Loki specifically within a Grafana-centric observability stack, rather than evaluating it purely as a standalone log store.

Loki vs. the Alternatives

Loki is not the only choice, and it isn't the right choice for every team. The Elastic Stack (Elasticsearch, Logstash or Beats, Kibana) remains the most flexible option for teams that genuinely need ad hoc, unpredictable full-text search across massive log corpora - security investigations and compliance audits often fall into this category, where you can't know in advance what you'll need to search for. That flexibility comes at the cost of significantly higher compute and storage requirements for equivalent log volume, along with a more complex licensing history (Elastic's move to the Server Side Public License and the subsequent OpenSearch fork by AWS in 2021 are worth knowing about if licensing terms matter to your organization).

OpenSearch inherits Elasticsearch's architecture and query flexibility under an Apache 2.0 license, making it a common choice for teams that want Elasticsearch-like capabilities without the licensing constraints. It carries the same operational cost profile as Elasticsearch, however, since it uses the same full-text indexing approach.

Splunk remains dominant in large enterprises, security operations centers, and regulated industries, offering a mature ecosystem (SPL query language, extensive app marketplace, strong compliance tooling) but at a licensing cost that scales with ingested data volume and often becomes the single largest line item in an observability budget at scale.

Datadog Logs and similar SaaS platforms (New Relic, Honeycomb for a more trace-centric take) trade operational ownership for cost - you pay per gigabyte ingested and indexed, with no infrastructure to run, which suits teams that want to avoid operating any of this themselves but accept vendor lock-in and per-GB pricing that can become expensive at high volume.

Where Loki tends to win is specifically the combination of cost sensitivity, existing Prometheus/Grafana investment, and log query patterns that are genuinely label-driven - "show me logs from this service in this namespace" rather than "find any log anywhere that mentions this arbitrary phrase." Teams doing security forensics, compliance search, or genuinely unpredictable ad hoc investigation across unlabeled content are usually better served by Elasticsearch, OpenSearch, or Splunk, even accounting for the higher cost.

Key Takeaways

Conclusion

Loki's central bet - that most log queries are filtered by known metadata rather than arbitrary free text - turns out to be true often enough that the cost savings are substantial for the teams whose workloads fit that pattern. Pairing label-indexed storage with cheap object storage and a query language deliberately modeled on PromQL makes Loki feel like a natural extension of a Prometheus-based observability stack rather than a bolted-on second system, and that operational coherence is arguably its biggest advantage over the alternatives.

That said, Loki is not a drop-in replacement for Elasticsearch, OpenSearch, or Splunk in every scenario. Teams that need genuinely unpredictable full-text search, deep security forensics, or complex cross-field aggregations should weigh those requirements honestly against Loki's design constraints before migrating. The right decision usually comes down to a candid assessment of your actual query patterns: if your engineers mostly search by service, environment, and time range and then read through matching lines, Loki will very likely save you money and operational overhead. If your use case genuinely requires searching for arbitrary content across an entire fleet without knowing the relevant labels in advance, a fully-indexed system remains the safer choice.

As with most infrastructure decisions, the tool should follow the query pattern, not the other way around - and understanding Loki's architecture deeply enough to predict how it will behave under your specific logging habits is the difference between a smooth, low-cost rollout and a painful re-architecture six months later.

References