paulserban.eu

Writing Edition

Paul Serban

Writing, snippets & book notes

Vol. 2026

August 18, 2023

The Observability Logs Layer: From Structured Events to Grafana Loki Dashboards

A practical guide to designing a logging pipeline with Morgan, Winston, Pino, and OpenTelemetry, shipped through Grafana Alloy into Loki, and queried with LogQL

Introduction

Logs are the oldest form of observability and, paradoxically, the hardest to get right at scale. Every engineer has written a console.log at 2 a.m. to chase down a production incident, and every engineer has also been burned by log volumes so large that the one useful line was buried under ten thousand irrelevant ones. The logs layer sits alongside metrics and traces as one of the three classic observability pillars, and it remains the most concrete: a log line is a discrete, timestamped record of something that happened, with enough context to reconstruct a decision or a failure after the fact.

This article walks through the logs layer as an engineering discipline rather than a checklist of tools. It starts with the fundamental unit - the discrete timestamped event - and moves through the Node.js logging ecosystem (Morgan, Winston, Pino), the emerging OpenTelemetry Logs specification, and a concrete shipping pipeline using Grafana Alloy into Grafana Loki. It closes with LogQL query patterns, dashboard design, and the pitfalls that quietly erode the value of a logging system over time. The goal is not to sell a particular stack but to give you the reasoning to choose and operate one well.

The Discrete Timestamped Event: What a Log Actually Is

At its core, a log entry is a structured fact: at time T, in process P, something E happened, with attributes A. This sounds trivial, but it is the single most important conceptual shift in modern logging - moving away from free-text strings toward structured, machine-parseable records. A line like Error: user 4213 failed payment, order 88213, code CARD_DECLINED is readable by a human, but a machine has to guess at the schema every time. The same event as structured JSON is unambiguous, queryable, and aggregable:

{
  "level": "error",
  "event": "payment_failed",
  "userId": 4213,
  "orderId": 88213,
  "reason": "CARD_DECLINED",
  "ts": "2026-08-28T09:12:03.441Z"
}

This distinction matters because logs are consumed by two very different audiences: humans doing ad hoc investigation, and machines doing aggregation, alerting, and correlation. A discrete event model supports both if you treat the log line as a record with a fixed set of fields (timestamp, severity, service name, trace context, message, and a bag of structured attributes) rather than as prose. The W3C and OpenTelemetry communities have converged on a similar shape: every log record has a timestamp, a severity number and text, a body, and a set of resource and log attributes. This is not an accident - it mirrors how relational and time-series systems already think about events, and it is what makes correlation with traces and metrics possible later.

The practical implication is that your logging strategy should be decided before you pick a library. Decide what a "log event" means in your system: is it one line per HTTP request, one line per business transaction, one line per state transition? Mixing granularities without a consistent event model is the root cause of most log-based debugging headaches, because engineers can no longer predict what fields will be present on a given line.

The Node.js Logging Landscape: Morgan, Winston, Pino, and Alternatives

Node.js has three loggers that dominate real-world usage, and they solve different problems. Morgan is an HTTP request logging middleware for Express; it is not a general-purpose logger, and it should not be treated as one. It exists to emit one line per HTTP request with configurable tokens (method, URL, status, response time), and its main risk is that teams sometimes stop there, believing request logging alone constitutes observability.

Winston is a general-purpose, highly configurable logger built around the idea of transports (console, file, HTTP, third-party sinks) and formats (JSON, simple, colorized, custom). Its flexibility is both its strength and its cost: Winston's formatting pipeline runs synchronously in the main thread by default, and heavy custom formatters can become a measurable CPU cost under high log volume. Winston is a reasonable default for services that need multiple output destinations or custom log shaping without operating a separate log-processing tier.

Pino takes the opposite philosophy: it is deliberately minimal and optimized for throughput, using asynchronous, low-overhead JSON serialization and pushing formatting and transport work out of the hot path via worker threads (pino.transport). Pino's benchmarks consistently show it outperforming Winston and Bunyan in raw lines-per-second, which matters in high-throughput services like API gateways or payment processors where logging overhead is not free. The trade-off is a smaller built-in feature set; Pino expects you to compose behavior through its transport and serializer ecosystem rather than configuring one monolithic object.

Beyond these three, Bunyan (Pino's spiritual predecessor, also JSON-first) and Log4js (a Log4j-inspired framework with hierarchical loggers and appenders) remain in use in older codebases. On the Python side, the standard library logging module combined with structlog for structured output plays the equivalent role, and in Go, zerolog and zap fill the high-performance structured logging niche. The lesson across ecosystems is consistent: pick a library that defaults to structured output, supports asynchronous or buffered writes, and lets you attach contextual fields (request ID, trace ID, tenant ID) without string concatenation.

// logger.ts - a Pino-based structured logger with trace correlation
import pino from "pino";
import { context, trace } from "@opentelemetry/api";

export const logger = pino({
  level: process.env.LOG_LEVEL ?? "info",
  timestamp: pino.stdTimeFunctions.isoTime,
  formatters: {
    level(label) {
      return { level: label };
    },
  },
  mixin() {
    const span = trace.getSpan(context.active());
    if (!span) return {};
    const { traceId, spanId } = span.spanContext();
    return { traceId, spanId };
  },
});

// usage in a request handler
export function handlePayment(orderId: string, userId: number) {
  logger.info(
    { event: "payment_attempt", orderId, userId },
    "processing payment",
  );
  try {
    // ... payment logic
    logger.info(
      { event: "payment_succeeded", orderId, userId },
      "payment completed",
    );
  } catch (err) {
    logger.error(
      { event: "payment_failed", orderId, userId, err },
      "payment failed",
    );
    throw err;
  }
}

The mixin hook above is the key pattern: every log line automatically carries the active OpenTelemetry trace and span ID, which is what makes it possible to pivot from a trace in a tracing backend directly to the corresponding log lines in Loki - a linkage that is otherwise easy to lose.

OpenTelemetry Logs and the Shift Toward a Vendor-Neutral Model

OpenTelemetry (OTel) began with traces and metrics, and its logging support matured later, but it has become the de facto neutral data model for all three signals. OpenTelemetry Logs defines a log record format with fields including Timestamp, ObservedTimestamp, SeverityNumber, SeverityText, Body, Attributes, Resource, and - critically - TraceId and SpanId for correlation. The design goal is explicit: logs should not be a separate, disconnected pillar but should share resource attributes and trace context with metrics and traces emitted from the same process.

In practice, most teams do not rewrite their application logging to use the OpenTelemetry Logs API directly. Instead, they keep Winston or Pino for application-level logging and use an OpenTelemetry Collector (or Grafana Alloy, which is compatible with the OpenTelemetry Collector's pipeline model) to receive, enrich, and export those logs alongside OTel-native traces and metrics. The OpenTelemetry Node SDK does provide log bridges - for example, @opentelemetry/instrumentation-winston and @opentelemetry/instrumentation-pino - that automatically inject trace context into log records emitted by these libraries, which is the mechanism behind the mixin pattern shown above, or can replace it depending on which layer you prefer to own the injection.

The alternatives worth naming here are Fluentd and Fluent Bit, which predate OpenTelemetry's logging support and remain widely deployed as log shippers, particularly in Kubernetes environments via the Fluent Bit DaemonSet pattern. Vector, written in Rust by Datadog (now part of the CNCF), is a newer high-performance alternative aimed at the same log-routing problem. Grafana's own Promtail was the original Loki-specific shipper but is now in maintenance mode, with Grafana Alloy positioned as its official replacement and the recommended path forward as of Loki's more recent releases.

Collection Agents: OpenTelemetry Collector, Grafana Alloy, and Alternatives

Between the application emitting a log line and the storage backend persisting it sits a piece of infrastructure that gets far less attention than it deserves: the collection agent. Its job sounds simple - read logs from disk or stdout, apply some transformation, and forward them somewhere - but at scale it becomes the component responsible for buffering during backend outages, enriching records with metadata the application doesn't have (Kubernetes pod labels, cloud region, node name), and doing so without falling over under a burst of traffic. Choosing an agent is an infrastructure decision with the same weight as choosing a database, and the field has consolidated around a handful of credible options with genuinely different architectural philosophies.

The OpenTelemetry Collector is the vendor-neutral reference implementation for this role, and it is built around three composable primitives: receivers, which ingest data (a filelog receiver for tailing files, an otlp receiver for data pushed directly over the OTLP protocol); processors, which transform data in flight (batching, attribute manipulation, sampling, redaction of sensitive fields); and exporters, which send the resulting data to one or more backends (loki, elasticsearch, clickhouse, otlp for chaining to another collector). Because logs, metrics, and traces all flow through the same pipeline abstraction, a single Collector deployment can carry all three signals with shared processing logic, which is precisely the point of OpenTelemetry as a project: one vendor-neutral data path instead of a different agent per signal per backend. The Collector is typically deployed in one of two topologies - as an agent running alongside every workload (a Kubernetes DaemonSet or sidecar) for local collection and enrichment, or as a gateway, a centralized fleet of Collector instances that agents forward to for heavier processing, tail-based sampling, or fan-out to multiple backends. Most production deployments use both tiers, keeping the per-node agent thin and pushing expensive work to the gateway layer where it can be scaled independently.

Grafana Alloy is best understood as Grafana Labs' own OpenTelemetry Collector distribution rather than a competing project: it embeds the OpenTelemetry Collector's component model and adds first-class components for Grafana's own backends (loki.write, prometheus.remote_write, otelcol.exporter.otlp for Tempo) alongside native Prometheus-style service discovery and relabeling, which is what powers the discovery.kubernetes and discovery.relabel components used in the Alloy pipeline shown earlier. Where it diverges from a stock Collector deployment is ergonomics for the Grafana stack specifically: label-based relabeling that maps directly onto Loki's model, and a single binary that replaced what used to be three separate Grafana agents (Grafana Agent for metrics, Promtail for logs, and the Grafana Agent traces mode), reducing the number of moving parts teams have to operate. For a team already standardized on Loki, Mimir, and Tempo, Alloy removes friction that a stock OpenTelemetry Collector configuration would otherwise require reconstructing manually.

The alternatives worth weighing seriously are Fluentd, Fluent Bit, and Vector, each with a different sweet spot. Fluentd, a CNCF graduated project, was for years the default Kubernetes logging agent, built around a plugin ecosystem (over a thousand community plugins for inputs, filters, and outputs) written primarily in Ruby with performance-critical paths in C; its plugin breadth is unmatched, but its memory footprint and per-event overhead are noticeably higher than newer alternatives, which matters when running as a DaemonSet on every node in a large cluster. Fluent Bit is Fluentd's lightweight sibling, written in C, with a fraction of the memory footprint and startup time, and has become the more common default for the DaemonSet role specifically because of that efficiency - it speaks a compatible plugin model but with a leaner core and fewer built-in transformation capabilities than Fluentd or the OpenTelemetry Collector. Vector, built in Rust and now part of the CNCF sandbox, was designed from the ground up around a strongly typed data model (its Vector Remap Language, VRL, for transformations) and is frequently cited for the best raw throughput-per-CPU-core among this group, with the trade-off that its ecosystem of pre-built integrations, while solid, is younger and smaller than Fluentd's.

Deciding between these is less about which is objectively fastest and more about where you already have operational gravity. Teams standardized on Grafana's LGTM stack (Loki, Grafana, Tempo, Mimir) gain the most from Alloy's tight integration and reduced component count. Teams that need broad backend flexibility, multiple output destinations, and a mature plugin catalog - feeding logs to more than one vendor, or handling long-tail legacy log formats - often keep Fluentd or Fluent Bit in place, sometimes running Fluent Bit as the node-level agent and forwarding to an OpenTelemetry Collector gateway for OTel-native processing. Vector earns its place in throughput-sensitive environments where CPU cost per gigabyte of logs is a real line item. And any team building toward a genuinely vendor-neutral stack - one that might swap Loki for ClickHouse or add a second backend later without re-instrumenting every service - should default to the OpenTelemetry Collector's receiver/processor/exporter model as the long-term investment, using Alloy or Fluent Bit as an implementation detail of that model rather than a permanent commitment.

Building the Pipeline: From Application to Grafana Alloy to Loki

Grafana Loki is a log aggregation system designed with a specific, opinionated trade-off: unlike Elasticsearch, it does not index the full text of log lines. Instead, Loki indexes only a small set of labels (service name, environment, pod name, and similar low-cardinality metadata) and stores the log content itself in compressed chunks. This makes Loki dramatically cheaper to run at scale than a full-text search index, at the cost of requiring queries to filter first by label, then by content - which is precisely the model LogQL is built around.

Grafana Alloy is Grafana Labs' OpenTelemetry Collector distribution, unifying what used to be separate agents (Grafana Agent, Promtail) into a single vendor-neutral binary that can collect logs, metrics, and traces and route them to Loki, Prometheus/Mimir, and Tempo respectively. Alloy uses a component-based configuration language (Alloy syntax, formerly River) where each component - a file-tailing source, a processor, an exporter - is wired together explicitly, which makes multi-stage log pipelines easier to reason about than Promtail's older YAML pipeline stages.

A minimal Alloy configuration for shipping container logs from Kubernetes to Loki looks like this:

// config.alloy - tail container logs and ship to Loki with label enrichment
discovery.kubernetes "pods" {
  role = "pod"
}

discovery.relabel "pod_logs" {
  targets = discovery.kubernetes.pods.targets
  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_label_app"]
    target_label  = "app"
  }
}

loki.source.kubernetes "pods" {
  targets    = discovery.relabel.pod_logs.output
  forward_to = [loki.process.parse_json.receiver]
}

loki.process "parse_json" {
  forward_to = [loki.write.default.receiver]

  stage.json {
    expressions = {
      level     = "level",
      event     = "event",
      trace_id  = "traceId",
    }
  }

  stage.labels {
    values = {
      level = "",
    }
  }
}

loki.write "default" {
  endpoint {
    url = "https://loki.internal.example.com/loki/api/v1/push"
  }
}

Two design decisions in this pipeline matter more than the syntax. First, label enrichment happens via discovery.relabel, keeping labels to a small, bounded set (namespace, app) rather than promoting every JSON field to a label - this is the single most important operational decision in a Loki deployment, discussed further below. Second, stage.json extracts structured fields from the log body for use in later filtering or as labels, but only level is promoted to a label; event and trace_id remain queryable content, searchable through LogQL's line filters and parsers without inflating the index.

Choosing a Storage Backend: Loki, Elasticsearch/OpenSearch, and ClickHouse

The storage backend is the decision that everything else in the logs layer is downstream of, because it determines what a query costs, what a schema change costs, and what your infrastructure bill looks like at ten times today's volume. The three backends teams actually choose between in practice are Grafana Loki, the Elasticsearch/OpenSearch family, and ClickHouse, and they represent three genuinely different bets on how logs should be indexed and stored, not just three brands of the same thing.

Loki's bet, as covered above, is that full-text indexing is a cost most teams don't need to pay. It indexes only labels and stores compressed log chunks in cheap object storage (S3, GCS, or equivalent), which keeps both storage and ingestion cost low and makes horizontal scaling straightforward because chunks are immutable and stateless to serve. The trade-off shows up at query time: a LogQL query without a well-chosen label selector has to decompress and grep through every matching chunk, which is fine for a bounded, label-scoped investigation but slow for exploratory full-text search across an entire fleet. Loki also has no native support for complex aggregations or joins across fields the way a search engine or columnar database does - its metric queries are powerful for counting and rating, but they are not a substitute for structured analytical queries.

Elasticsearch and its open-source fork OpenSearch take the opposite bet: full-text indexing of every field, by default, using an inverted index built on Apache Lucene. This makes ad hoc search extremely fast and flexible - free-text queries, fuzzy matching, and complex boolean queries across arbitrary fields all work well out of the box, and the Elastic Common Schema (ECS) or OpenSearch's equivalent conventions give teams a standard field vocabulary to build dashboards and detections against. The cost is operational and financial: indexing every field means significantly higher CPU, memory, and disk usage per gigabyte of raw log data than Loki, and Elasticsearch clusters require real operational investment - shard sizing, index lifecycle management (ILM/ISM) to roll over and delete old indices, and careful capacity planning to avoid the cluster going yellow or red under load. Elasticsearch and OpenSearch remain the right choice when search flexibility and mature ecosystem tooling (Kibana, OpenSearch Dashboards, SIEM use cases) matter more than raw cost efficiency, particularly for security and compliance logging where analysts genuinely need arbitrary full-text search across unpredictable fields.

ClickHouse represents a third position that has gained significant traction for logs in recent years: a columnar, SQL-native OLAP database that was not originally built for logs but turns out to be extremely well suited to them. Because ClickHouse stores each column separately and compresses aggressively, queries that filter or aggregate on a handful of columns (timestamp, service name, status code) scan only that data rather than entire rows, giving it both fast filtered queries and fast aggregations without Elasticsearch's per-field indexing overhead. Tools like Grafana's own connector for ClickHouse, and purpose-built platforms such as SigNoz and the open-source clickhouse-quickwit-style projects, have popularized ClickHouse as a logs backend precisely because it offers SQL query flexibility (joins, window functions, arbitrary aggregations) at a fraction of Elasticsearch's storage cost, and better raw ingestion throughput than either alternative for high-cardinality structured data. The trade-off is that ClickHouse has no built-in full-text search comparable to Lucene's inverted index - text search relies on LIKE, match(), or trigram/bloom-filter secondary indices, which are effective for known patterns but weaker for genuinely open-ended free-text exploration.

Choosing between the three is less about which is "best" and more about matching the backend to your actual query pattern. If most log access is scoped, label-filtered incident investigation - "show me errors from this service in this time window" - Loki's model fits and its cost profile wins. If analysts need unpredictable, cross-field full-text search, especially for security or compliance use cases, Elasticsearch or OpenSearch earns its operational overhead. If your team already thinks in SQL and needs both high-throughput structured ingestion and flexible analytical queries - joining logs against a dimension table of deployments, for instance - ClickHouse is increasingly the pragmatic middle ground. Several teams run more than one: Loki or ClickHouse for the high-volume operational firehose, with a smaller Elasticsearch/OpenSearch cluster reserved for security logs where full-text search is non-negotiable.

LogQL Basics: Querying Logs as Streams

LogQL is Loki's query language, deliberately modeled after PromQL so that anyone comfortable with Prometheus queries can pick it up quickly. Every LogQL query starts with a log stream selector - a label matcher in curly braces - which is mandatory and determines which compressed chunks Loki has to scan at all.

{namespace="payments", app="checkout-api"}

This alone returns raw log lines matching those labels. From there, LogQL layers on line filters, which are simple substring or regex matches applied to the raw text before any parsing:

{namespace="payments", app="checkout-api"} |= "payment_failed"

The |= operator means "line contains"; !=, |~, and !~ provide negative and regex variants. Because line filtering happens on the raw text without deserializing JSON, it is cheap and should be your first filter whenever possible. After filtering, LogQL supports parser expressions - | json, | logfmt, | pattern - which extract structured fields from the log body into labels usable downstream:

{namespace="payments", app="checkout-api"}
  |= "payment_failed"
  | json
  | reason = "CARD_DECLINED"
  | line_format "{{.userId}} order={{.orderId}} reason={{.reason}}"

This chain filters to lines containing "payment_failed", parses the JSON body, filters further on the extracted reason field, and reformats the output line for readability. LogQL's real power, though, comes from its metric queries, which aggregate log streams into time series using functions like rate(), count_over_time(), and sum by (...):

sum by (reason) (
  count_over_time(
    {namespace="payments", app="checkout-api"} |= "payment_failed" | json
    [5m]
  )
)

This produces a time series of failure counts grouped by decline reason, which can be dropped straight into a Grafana panel or fed into an alert rule via Loki's ruler component - turning unstructured investigation into a first-class monitoring signal without duplicating the same information into a separate metrics pipeline.

Patterns, Practices, Pitfalls, and Anti-Patterns

The most consequential mistake in any Loki deployment is high-cardinality labeling. Loki creates a separate stream - and separate chunk storage - for every unique combination of label values, so promoting a field like userId, orderId, or requestId to a label rather than keeping it in the log body can multiply the number of streams by orders of magnitude, degrading both ingestion and query performance and inflating storage costs. The rule of thumb from Grafana's own documentation is to keep labels to values with low, bounded cardinality - service name, environment, region, log level - and let LogQL's parsers extract high-cardinality fields from the body at query time instead.

A second common anti-pattern is inconsistent event schemas within the same stream. If half your payment_failed events include a reason field and half don't, or if the field is sometimes a string and sometimes a nested object, every downstream LogQL query and dashboard panel becomes conditional logic instead of a straightforward filter. Enforcing a schema - even informally, via a shared TypeScript interface or JSON Schema validated in CI - pays for itself the first time someone needs to build an alert on that field.

A related pitfall is logging at the wrong level of granularity: either too sparse to reconstruct what happened (a single "request completed" line with no context) or so verbose that meaningful signals drown in noise (debug-level logs left enabled in production, one line per database call). A useful practice is to log business events, not implementation details - record order_placed, payment_failed, inventory_reserved rather than entering function processOrder or SQL query executed. Combine this with sampling for genuinely high-volume, low-value events (health checks, successful cache hits) rather than disabling logging wholesale, since sampling gives you a statistically representative view without full storage cost.

Silent log loss is another operational trap: if your shipper (Alloy, Fluent Bit) cannot keep up with burst traffic and drops or blocks, you can lose exactly the log lines you need most, during an incident. Configuring backpressure handling, on-disk buffering, and monitoring the shipper's own health metrics (queue depth, dropped record counts) is not optional in a production deployment - treat your logging pipeline itself as a service with an SLO.

Finally, teams sometimes conflate logs with metrics or traces, using count_over_time queries in Loki as a substitute for a proper metrics pipeline. This works at small scale but degrades as volume grows, because Loki's index is optimized for label-based retrieval, not high-frequency numerical aggregation. If a signal genuinely needs sub-minute resolution or complex mathematical operations across dimensions, it likely belongs in Prometheus or Mimir as a counter or histogram, with logs reserved for the qualitative "what happened and why" context that metrics cannot capture.

Dashboards and Panels: Making Logs Actionable

A Grafana dashboard built on Loki data typically combines two panel types: log panels showing raw or formatted lines for investigation, and time series panels driven by LogQL metric queries for trend detection. The design principle that separates a useful dashboard from a noisy one is the same principle that governs good logging itself - start broad, then narrow. A well-designed operational dashboard leads with an aggregate error-rate panel (sum(count_over_time({app="checkout-api"} |= "level=error" [5m]))), lets an engineer click through to a filtered log panel scoped to the same time range and labels, and only then exposes raw log lines for deep investigation. This mirrors the same drill-down flow that a trace waterfall provides for latency problems.

Panel design should also respect the cost model discussed earlier: a dashboard with a dozen high-cardinality count_over_time queries refreshing every ten seconds can generate meaningful load on a Loki cluster, particularly if those queries scan long time ranges without label pre-filtering. Grafana's explore view is well suited for ad hoc investigation, where a broad, unindexed query is acceptable because a human is iterating on it, but persistent dashboards should be built from queries that have been validated for performance, ideally with recording rules in Loki's ruler pre-computing expensive aggregations so dashboard panels read from cheap, pre-aggregated series rather than recomputing them on every load.

A practical dashboard for the payments example above might include: an error-rate-over-time panel by reason, a top-N table of failing orderId values extracted via | json | reason != "", a log panel filtered to level="error" for the selected time range, and an annotation layer pulling deploy events so error spikes can be correlated with recent releases. This combination turns the logs layer from a passive record into an active diagnostic tool that answers "what changed" as readily as "what broke."

Panel-Level Practices Worth Standardizing

A few panel-level conventions separate dashboards that stay useful for years from ones that quietly rot within a quarter. Every panel should carry an explicit, human-readable title that states what it measures and its unit, rather than leaving viewers to reverse-engineer intent from a raw LogQL expression - "Payment Decline Rate by Reason (per 5m)" tells an on-call engineer at 3 a.m. what they're looking at faster than the query itself ever could. Panels that share the same underlying stream selector should also share consistent time-range and variable behavior, typically by wiring them to Grafana dashboard variables (for example a $namespace or $service template variable) rather than hardcoding labels into each query, so a single dropdown re-scopes the entire dashboard instead of forcing an engineer to edit ten queries by hand during an incident. Color and threshold conventions matter more than they seem: reserving red consistently for error-related series and amber for warning-level aggregates, across every dashboard in an organization, lets engineers pattern-match severity at a glance even on unfamiliar dashboards, which meaningfully speeds up triage when someone outside the owning team is paged in.

Legends and units deserve the same discipline. A count_over_time panel should label its Y-axis and legend with the actual unit (failures per 5 minutes, not just "value"), and legends on multi-series panels (one line per reason, for instance) should be sorted or truncated sensibly so a panel with twenty decline reasons doesn't render an unreadable wall of color. It's also worth setting sensible default time ranges per dashboard purpose: an incident-response dashboard defaulting to the last hour surfaces what's happening now, while a capacity-planning dashboard built on the same underlying streams might default to seven or thirty days, and conflating the two defaults on one dashboard tends to produce a view that serves neither use case well.

Alerting Patterns Built on Top of Dashboards

Dashboards and alerts should be built from the same underlying LogQL queries rather than maintained as parallel, drifting definitions of "broken". When a panel's aggregate query (sum by (reason) (count_over_time(...))) is also the basis for a Loki ruler alert rule, a dashboard viewer investigating a page can trust that what triggered the alert is exactly what they're looking at, rather than a subtly different threshold defined elsewhere. A common and effective pattern is layering alert severity: a lower-severity, higher-sensitivity alert that pages a Slack channel for early signal, paired with a higher-severity, higher-confidence alert (typically requiring a sustained condition over a longer window via LogQL's range vector, such as [15m] instead of [5m]) that pages on-call directly, reducing alert fatigue from single noisy spikes while still catching genuine sustained degradation quickly.

Runbook links embedded directly in alert annotations are a small addition with outsized payoff: an alert that fires with a link to both the relevant dashboard (pre-filtered to the alerting labels) and a runbook describing known causes and first response steps turns a 2 a.m. page from a cold investigation into a guided procedure. It's also worth periodically auditing alert-to-dashboard coverage in the other direction - for every alert rule defined in Loki's ruler, confirming a corresponding dashboard panel exists that visualizes the same signal over time, since an alert with no visual backing makes it hard for a responder to judge trend, severity, or whether the issue is already recovering.

Key Takeaways

Five practical steps to apply this immediately:

Conclusion

The logs layer has not become obsolete in the age of distributed tracing and high-cardinality metrics - it has become more precise. The shift from unstructured strings to discrete, structured, timestamped events with trace correlation is what makes logs a genuine peer to metrics and traces rather than a fallback when those signals run out. Tools like Pino and the OpenTelemetry logging bridges make structured, low-overhead logging the path of least resistance in application code, while Grafana Alloy and Loki make it economically viable to store and query that data at scale by trading full-text indexing for disciplined labeling. LogQL then lets that discipline pay off, turning a stream of JSON lines into both ad hoc investigative queries and first-class monitoring signals.

None of this removes the need for engineering judgment. The libraries and query languages are means, not ends; the actual skill being exercised is deciding what constitutes a meaningful event in your system, what belongs in a label versus a body, and what belongs in a log at all versus a metric or a trace. Teams that get this right end up with a logs layer that scales gracefully and remains genuinely useful during an incident - which, after all, is the only real measure of an observability system's worth.

References

A quick flashcard preview - one example of each question type in this post. This is a demo of what you'll find in the full quiz app.

Flashcard preview - 1 / 9Multiple Choice

multiple choice - intermediate - auto-graded

A team's dashboard alert misses some payment_failed events because the reason field is sometimes present as a string and sometimes missing entirely across different code paths. What anti-pattern does this describe?

Choose an answer