paulserban.eu

Writing Edition

Paul Serban

Writing, snippets & book notes

Vol. 2026

August 03, 2023

Observability Engineering: How Instrumentation, Telemetry, Metrics, Logs, and Alerts Actually Fit Together

A practitioner's map from raw signals to decisions - connecting instrumentation, the three pillars of observability, alerting, analytics, tracking, and KPIs into one coherent system

Introduction

Every production system generates a constant stream of signals: a request arrives, a database query runs slow, a queue backs up, a deploy ships, a user abandons a checkout flow. The engineering challenge is not generating these signals - modern systems produce far more data than any human can read - but turning them into something a team can act on within seconds or minutes of a problem appearing. That translation, from raw system exhaust into actionable understanding, is the actual job of observability tooling, and it is built from a stack of related but distinct concepts: instrumentation, telemetry, metrics, logs, traces, alerts, analytics, tracking, and KPIs.

These terms get used almost interchangeably in casual conversation, which causes real damage in practice. Teams buy a metrics platform and call it "observability", or build a dashboard full of KPIs that nobody uses during an incident, or instrument everything they can think of and then drown in cardinality bills. This article works through each concept individually, then - more importantly - through how they connect into a single pipeline: instrumentation produces telemetry, telemetry is shaped into metrics/logs/traces, those feed alerts and analytics, and analytics get distilled into KPIs that leadership and engineers both use to make decisions. Along the way there are working code examples in TypeScript and Python, and a set of trade-offs that experienced teams learn the hard way.

Context and Problem Overview: Why Systems Need Observability

For a long time, "monitoring" was enough. A single application server, a handful of known failure modes, and a dashboard of CPU, memory, and request-count graphs could tell an operator most of what they needed to know. Monitoring in this sense answers questions you already thought to ask in advance - it works well against known-unknowns: things you expect might fail, so you build a check for them. The shift to distributed systems, microservices, containers, and managed infrastructure broke this model. A single user request might now traverse a dozen services, three data stores, a message queue, and a third-party API, each independently deployed and independently scaled. Failures increasingly emerge from the interaction between components rather than from any single component being unhealthy - a class of problem often called unknown-unknowns, because nobody wrote a dashboard for it in advance.

This is the gap that the term "observability" was created to describe. The word itself comes from control theory, where Rudolf Kálmán defined it in the 1960s as a measure of how well a system's internal state can be inferred from its external outputs. Applied to software, a system is observable if engineers can understand what is happening inside it - including conditions nobody explicitly anticipated - purely by examining the data it emits, without having to ship new code or attach a debugger to investigate. The Google SRE Book and the more recent Observability Engineering (O'Reilly, 2022) both draw a similar distinction: monitoring tells you whether something is wrong against a predefined threshold, while observability gives you the raw material to ask arbitrary new questions when something wrong wasn't anticipated at all.

This distinction has real organizational consequences. Teams that only build monitoring dashboards for known failure modes tend to get blindsided by novel incidents and then spend hours adding ad-hoc logging mid-incident to understand what happened - precisely when time matters most. Teams that invest in genuine observability, meaning rich, high-cardinality, queryable telemetry, can instead interrogate the live system with new questions during an incident: "show me every request from this customer ID that touched service X and took longer than 2 seconds in the last hour", for example. That capability doesn't come from a single tool; it comes from disciplined instrumentation, a coherent telemetry pipeline, and platforms designed to correlate signals rather than silo them.

Core Concepts: Instrumentation, Telemetry, and the Three Pillars

Instrumentation is the code-level work of making a system emit information about what it's doing: timers around a function call, a counter incremented on every cache miss, a log line written when a payment fails, a span created around an outbound HTTP call. Instrumentation can be manual, where a developer explicitly writes the calls that emit data, or automatic, where an agent or library patches common frameworks (an HTTP client, a database driver, a web framework) to emit telemetry without the developer writing extra code. Most real systems use both: automatic instrumentation for broad, consistent baseline coverage, and manual instrumentation for business-specific logic that a generic agent could never know to measure. The industry has converged strongly on OpenTelemetry, a CNCF project, as the vendor-neutral standard for this layer - it defines a common API and SDK for creating metrics, logs, and traces, and a wire protocol (OTLp) for shipping them, so that instrumentation code doesn't need to be rewritten every time a team switches observability backends.

Telemetry is simply the data that instrumentation produces - the term, borrowed from remote sensing and aerospace engineering, refers to measurements transmitted from a distance for monitoring purposes. In software, telemetry is conventionally organized into three pillars, each optimized for a different kind of question. Metrics are numeric measurements aggregated over time - cheap to store, cheap to query, excellent for trends and thresholds, but they lose per-event detail once aggregated. Logs are discrete, timestamped records of individual events - rich in detail and context, but expensive to store at scale and harder to aggregate meaningfully across a fleet. Traces represent the path of a single request as it moves through a distributed system, made up of spans that record where time was spent and which services were involved - ideal for understanding latency and causality across service boundaries, but the most complex to implement correctly. None of the three pillars is sufficient alone; the craft of observability engineering is largely about knowing which pillar answers which question, and building the connective tissue that lets an engineer move fluidly between them.

Deep Technical Explanation: How Metrics, Logs, Traces, and Context Propagation Work Together

Metrics are built from a small set of primitives that show up in nearly every metrics system, including Prometheus, the de facto standard for infrastructure metrics in cloud-native environments. A counter only increases (total requests served, total errors), a gauge can go up or down (current queue depth, active connections), and a histogram or summary buckets observations to let you later compute percentiles like p50, p95, or p99 latency. Metrics are usually attached to labels or dimensions - route, status_code, region - which let you slice a single metric many ways. This is also where the biggest operational risk in metrics systems lives: cardinality. Every unique combination of label values creates a new time series, and systems that attach unbounded values as labels - a raw user ID or a full request path with embedded IDs - can silently create millions of time series, degrading query performance and inflating storage costs far beyond what the team intended.

Logs capture what happened at a specific point in time, in as much detail as the developer chose to include. The industry has moved decisively from unstructured, free-text log lines toward structured logging, where each entry is emitted as a JSON object (or similar) with consistent fields - timestamp, severity, service name, and a message - rather than a sentence meant only for human eyes. Structured logs are dramatically easier to index, filter, and aggregate in log platforms like the Elastic Stack or Grafana Loki. A second critical practice is correlation: attaching a request ID or, better, a distributed trace ID to every log line written during that request's lifecycle. Without this, an engineer investigating an incident has to manually stitch together log lines from a dozen services using timestamps and guesswork; with it, every log line related to a single problematic request can be pulled up with one query.

Traces solve a different problem: understanding causality and latency across service boundaries. A trace is a tree of spans, where each span represents a unit of work - an HTTP handler, a database call, a downstream RPC - with a start time, duration, and parent-child relationship to other spans. The mechanism that makes distributed tracing possible across independently deployed services is context propagation: as a request crosses a network boundary, the trace's identifying information travels with it, typically as HTTP headers. The W3C Trace Context specification standardized this as the traceparent and tracestate headers, which most modern tracing libraries, including OpenTelemetry's, now implement by default. Because capturing a full trace for every single request at high volume is often prohibitively expensive, teams apply sampling - head-based sampling decides whether to record a trace at its start (simple, but risks dropping the rare slow or erroring request you actually care about), while tail-based sampling buffers spans and decides after the fact, keeping traces that were slow or contained errors even if most ordinary traces get discarded.

The real payoff of these three pillars appears when they're correlated rather than treated as separate tools. A metric dashboard shows p99 latency spiking on the checkout service; an engineer clicks into an "exemplar" - a sampled trace ID attached to that specific metric data point - and jumps directly into a trace showing exactly which downstream call was slow; from that trace, the same trace ID pulls up every structured log line emitted during that exact request. This correlated-navigation pattern, supported natively by platforms like Grafana (metrics, Loki logs, and Tempo traces sharing exemplars) and by commercial tools like Honeycomb and Datadog, is what separates a genuinely observable system from a pile of disconnected dashboards. It is also the strongest practical argument for standardizing instrumentation on OpenTelemetry from the start, since hand-rolled, inconsistent instrumentation rarely produces the shared identifiers this correlation depends on.

From Raw Telemetry to Meaning: Alerts, Analytics, Tracking, and KPIs

Alerts are the mechanism that turns passive telemetry into an active notification that a human needs to look at something now. Technically, an alert is a rule evaluated continuously against metrics or log-derived signals - "fire if error rate exceeds 5% for five minutes", for instance - that triggers a notification through a routing system such as Prometheus's Alertmanager or a commercial incident-management tool like PagerDuty. The Google SRE Workbook popularized a more disciplined approach to this: rather than alerting on arbitrary thresholds for every metric, define Service Level Objectives (SLOs) around what actually matters to users - availability, latency - and alert on the rate at which you are burning your error budget, using multi-window, multi-burn-rate alerts that catch both fast, severe outages and slow, sustained degradations without paging someone for every minor blip. Alerting quality matters enormously because it is the one part of the pipeline with a direct cost in human attention; badly tuned alerts don't just waste time, they train engineers to ignore pages, which is far more dangerous than having no alert at all.

Analytics and tracking occupy an adjacent but distinct space, usually oriented toward product and business behavior rather than infrastructure health. Tracking refers to the practice of capturing discrete events tied to user or business actions - "user clicked checkout", "trial started", "email opened" - typically defined in advance through a tracking plan so that event names and properties stay consistent across a codebase and across teams. Analytics is what happens to that tracked data afterward: computing funnels, retention curves, cohort behavior, and feature adoption using platforms like Amplitude, Mixpanel, or Segment (which itself often functions as a routing layer that fans tracked events out to multiple downstream analytics tools). The overlap with infrastructure telemetry is real - both are, structurally, event streams - but the audience and cadence differ: infrastructure telemetry is consumed in near-real-time by engineers during operations, while product analytics is more often consumed in aggregate, over days or weeks, by product managers and growth teams making roadmap decisions.

KPIs - Key Performance Indicators - are the layer where both worlds get compressed into a small number of numbers that leadership actually tracks over time. On the engineering side, the DORA metrics (deployment frequency, lead time for changes, change failure rate, and time to restore service), popularized through the "Accelerate" research program and book, have become a widely adopted standard for measuring software delivery performance rather than raw activity. Alongside them sit reliability KPIs like mean time to recovery (MTTR) and error-budget burn rate, and product KPIs like activation rate or monthly active users. The critical engineering discipline here is traceability: a good KPI should be explainable in terms of the metrics, logs, and traces underneath it, so that when a KPI moves in the wrong direction, someone can drill down through the pipeline - KPI, to analytics, to raw telemetry - and find the actual cause rather than treating the number as a black box.

Implementation Walkthrough: Instrumenting a Service in Practice

The concepts above become much more concrete with code. The examples below show a small but realistic slice of a Node.js/TypeScript order service instrumented with OpenTelemetry for tracing, a Python worker instrumented with prometheus_client for metrics following the RED method (Rate, Errors, Duration - a pattern popularized by Tom Wilkie at Weaveworks for request-driven services), and a structured logger correlated to the active trace. None of these snippets are toy examples stripped of realistic detail; they follow patterns you'd actually find in a production codebase, including error handling and label hygiene to avoid cardinality blowups.

The connecting idea across all three snippets is that instrumentation should be woven into existing control flow - middleware, decorators, context managers - rather than scattered as one-off calls, because that's the only way it stays consistent as the codebase grows and as new engineers touch the code without deep observability expertise.

// order-service/src/tracing.ts
// Manual span creation around a downstream call, using OpenTelemetry's API.
import { trace, SpanStatusCode, context } from '@opentelemetry/api';

const tracer = trace.getTracer('order-service', '1.4.0');

export async function chargePayment(orderId: string, amountCents: number) {
  return tracer.startActiveSpan('payment.charge', async (span) => {
    // Keep labels/attributes low-cardinality-safe where they'll later be
    // mirrored into metrics; order IDs are fine on a span, but should
    // never become a metric label.
    span.setAttribute('order.id', orderId);
    span.setAttribute('payment.amount_cents', amountCents);

    try {
      const response = await fetch('https://payments.internal/charge', {
        method: 'POST',
        headers: { 'content-type': 'application/json' },
        body: JSON.stringify({ orderId, amountCents }),
      });

      if (!response.ok) {
        throw new Error(`payment service responded ${response.status}`);
      }

      span.setStatus({ code: SpanStatusCode.OK });
      return await response.json();
    } catch (err) {
      // Recording the exception keeps the full stack/message on the span
      // itself, so it shows up when a teammate opens this trace later.
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR, message: (err as Error).message });
      throw err;
    } finally {
      span.end();
    }
  });
}

// order-service/src/logger.ts
// Structured logging correlated to the active OpenTelemetry span.
import pino from 'pino';
import { trace } from '@opentelemetry/api';

const baseLogger = pino({ base: { service: 'order-service' } });

export function log(level: 'info' | 'warn' | 'error', message: string, fields: Record<string, unknown> = {}) {
  const activeSpan = trace.getActiveSpan();
  const spanContext = activeSpan?.spanContext();

  baseLogger[level]({
    ...fields,
    trace_id: spanContext?.traceId,
    span_id: spanContext?.spanId,
  }, message);
}

# worker/metrics.py
# RED-method instrumentation for a background job processor using
# prometheus_client. Labels are kept to a small, bounded set of values.
import time
from functools import wraps
from prometheus_client import Counter, Histogram

JOBS_TOTAL = Counter(
    "worker_jobs_total", "Total jobs processed", ["job_type", "status"]
)
JOB_DURATION_SECONDS = Histogram(
    "worker_job_duration_seconds", "Job processing duration", ["job_type"]
)

def instrumented_job(job_type: str):
    def decorator(fn):
        @wraps(fn)
        def wrapper(*args, **kwargs):
            start = time.perf_counter()
            status = "success"
            try:
                return fn(*args, **kwargs)
            except Exception:
                status = "error"
                raise
            finally:
                duration = time.perf_counter() - start
                JOB_DURATION_SECONDS.labels(job_type=job_type).observe(duration)
                JOBS_TOTAL.labels(job_type=job_type, status=status).inc()
        return wrapper
    return decorator

@instrumented_job(job_type="send_invoice_email")
def send_invoice_email(order_id: str) -> None:
    ...  # actual job logic

# prometheus/alerts/checkout-slo.yaml
# Multi-window, multi-burn-rate alert on an availability SLO, following
# the pattern described in the Google SRE Workbook's alerting chapter.
groups:
  - name: checkout-availability-slo
    rules:
      - alert: CheckoutFastBurn
        expr: |
          (
            sum(rate(worker_jobs_total{job_type="checkout", status="error"}[5m]))
            /
            sum(rate(worker_jobs_total{job_type="checkout"}[5m]))
          ) > (14.4 * 0.001)
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "Checkout error budget burning fast (14.4x over 5m)"

Trade-offs and Common Pitfalls

The most common and most expensive mistake teams make is treating cardinality as free. Every unique label combination on a metric, or every unique field value indexed in a log platform, creates additional series or index entries that the backend must store and query. A well-intentioned engineer adding a user_id or order_id label directly to a Prometheus metric can, on a busy service, generate millions of new time series within hours, and the resulting bill or query slowdown often lands on a completely different team than the one that wrote the code. The general rule that experienced teams converge on is: unbounded, high-cardinality identifiers belong on trace spans and structured log fields, where they're inherently designed for high-cardinality lookups, and only low-cardinality, bounded dimensions (route templates, status code classes, region names) belong as metric labels.

Sampling and instrumentation overhead present a related but distinct trade-off. Recording a full trace for every request at high throughput can add measurable latency and cost, so teams reach for sampling - but naive head-based sampling at a fixed rate (say, 1%) can systematically miss the rare, slow, or erroring requests that matter most for debugging, precisely because those are outliers in a sea of "normal" traffic. Tail-based sampling addresses this but requires buffering spans in a collector before deciding what to keep, adding operational complexity and memory pressure of its own. Similarly, dense manual instrumentation (spans and logs on every function call) can add real CPU and I/O overhead to hot code paths; the discipline is to instrument at meaningful boundaries - service entry points, external calls, expensive operations - rather than everywhere uniformly.

Beyond the technical trade-offs, there are organizational failure modes that are arguably more damaging because they're harder to detect. Alert fatigue develops when too many low-value alerts fire, training on-call engineers to acknowledge and dismiss pages reflexively rather than investigate them - a well-documented pattern that undermines the entire purpose of alerting. Vanity KPIs are the product-analytics equivalent: metrics that look good in a slide deck (raw signups, page views) but don't correlate with anything the business actually needs to improve, often because nobody validated that the KPI causally tracks the outcome it's supposed to represent. And tool sprawl - a metrics platform from one vendor, a log platform from another, a separate APM tool, and a separate product-analytics suite, none of them sharing identifiers - quietly recreates the exact silos that observability was supposed to eliminate, forcing engineers back into manual timestamp-matching across browser tabs during an incident.

Best Practices for Sustainable Observability

Several practices consistently separate teams that get real value from their observability investment from teams that accumulate dashboards nobody trusts. Standardizing instrumentation on OpenTelemetry from the outset avoids vendor lock-in and, more importantly, guarantees that trace IDs, span IDs, and resource attributes are consistent across every service regardless of which team wrote it - which is the precondition for the cross-pillar correlation described earlier. Alongside this, defining a small, explicit taxonomy for metric labels and log fields - decided once, documented, and enforced through code review or lint rules - prevents the slow, distributed decision-making that leads to cardinality explosions, since no single engineer adding one label to one metric ever feels like they're causing a systemic problem.

On the alerting and KPI side, the strongest pattern is grounding alerts in user-facing SLOs rather than arbitrary infrastructure thresholds, and periodically auditing both dashboards and alerts to retire ones nobody has looked at during an actual incident in the last quarter. This pairs well with a genuinely blameless incident-review culture, where postmortems focus on what the telemetry did or didn't reveal and what instrumentation gap to close next, rather than assigning individual fault - a practice that both the Google SRE Book and much of the wider SRE community treat as foundational rather than optional. Finally, KPIs should be reviewed for whether they still trace cleanly back to underlying telemetry; a KPI that leadership tracks but that engineers can no longer explain in terms of concrete metrics or events has effectively become disconnected from the system it claims to represent, and should either be re-derived or retired.

Mental Models and Analogies

A useful way to hold all of this in your head is to think of a system the way you'd think about a car. Metrics are the dashboard gauges - speed, fuel level, engine temperature - cheap to glance at, good for noticing that something is trending in the wrong direction, but they tell you nothing about why the temperature is rising. Logs are the car's black-box event recorder, capturing discrete, detailed events ("driver braked hard at 14:03:02", "check-engine code P0420 triggered") that you'd pull up after something specific happened. Traces are closer to a GPS breadcrumb trail with timestamps at each turn, showing you exactly which leg of the journey took longer than expected and in what order things happened relative to each other. None of these three replace the others; a mechanic diagnosing an intermittent problem needs all three, and a driver checking on a normal commute usually only needs the dashboard.

Analytics, tracking, and KPIs sit one level further out, closer to a compass than a speedometer. A speedometer (a raw metric) tells you your exact velocity right now; a compass (a KPI like deployment frequency or activation rate) tells you whether you're generally heading in the direction the business wants to go, aggregated over a much longer time horizon and much noisier underlying data. Neither view is more "correct" than the other - they answer different questions at different timescales, and confusing them is a common source of frustration, such as demanding that a KPI (a compass reading) explain a five-minute latency spike (a speedometer event), when the KPI simply wasn't built to resolve at that resolution.

Finally, it helps to picture the entire stack as a funnel rather than a set of separate tools: instrumentation at the wide top produces raw telemetry, which narrows into curated metrics/logs/traces, which narrows further into a handful of alerts and dashboards, which narrows again into the small set of KPIs a leadership team actually reviews weekly. Each stage of the funnel should be traceable back to the one below it - an executive should be able to ask "why did this KPI move" and, through a small number of hops, land on the actual raw events that caused it. When that traceability breaks down at any stage, the funnel stops being observability and becomes a set of disconnected reports.

The 80/20 of Observability

Given how large this space can get, it's worth naming the small set of practices that produce most of the practical value, because most teams don't need - and can't afford - to instrument everything to an equally exhaustive degree. The single highest-leverage decision is standardizing on OpenTelemetry for instrumentation and context propagation from day one; everything else in this article - correlation between pillars, exemplars, multi-service tracing - depends on having consistent trace and span identifiers flowing through the system, and retrofitting that consistency onto a codebase with years of inconsistent, hand-rolled instrumentation is far more expensive than adopting it early.

The second highest-leverage practice is applying the RED method (rate, errors, duration) to every request-driven service and the USE method (utilization, saturation, errors), described by Brendan Gregg, to every resource - CPU, memory, disk, queues - rather than trying to enumerate every possible metric a system could theoretically expose. These two small checklists cover the large majority of what's needed to detect that something is wrong and roughly where, even though they intentionally don't cover every business-specific signal. Layered on top of that, a handful of SLO-based alerts on user-facing latency and error rate, plus structured, trace-correlated logging on error paths, typically catches the overwhelming majority of incidents a team will actually face - with everything more elaborate (tail-based sampling, custom business dashboards, deep product analytics) delivering real but comparatively smaller marginal value.

Key Takeaways

Conclusion

Instrumentation, telemetry, metrics, logs, traces, alerts, analytics, tracking, and KPIs are not competing tools to choose between - they're stages in a single pipeline that starts with a line of code recording that something happened and ends with a leadership team deciding what to build next. The teams that get real, durable value from this stack are rarely the ones with the most dashboards; they're the ones who kept the pipeline coherent end to end, so that a number moving on an executive scorecard can, in a few clicks, be traced all the way back down to the specific request, span, and log line that explains it.

Where this space is heading next is a continued push toward lower-overhead, higher-fidelity telemetry: eBPF-based automatic instrumentation that captures traces and metrics from the kernel without requiring code changes at all, wider adoption of OpenTelemetry as the default rather than the exception, and AI-assisted anomaly detection layered on top of existing telemetry to surface unknown-unknowns before a human would have thought to look. None of that changes the underlying architecture described in this article, though - it only makes each stage of the funnel cheaper and faster to build, which is exactly why getting the fundamentals right now continues to compound in value.

If there's a single principle to carry forward, it's this: build for the question you haven't thought to ask yet, not just the dashboard you need today. That's the actual difference between monitoring and observability, and it's the difference that matters most at 3 a.m. during an incident nobody predicted.

References

A quick flashcard preview - one example of each question type in this post. This is a demo of what you'll find in the full quiz app.

Flashcard preview - 1 / 9Multiple Choice

multiple choice - advanced - auto-graded

Why does defining alerts around Service Level Objectives (SLOs) and error-budget burn rate generally produce better outcomes than alerting on arbitrary infrastructure thresholds?

Choose an answer