paulserban.eu

Writing Edition

Paul Serban

Writing, snippets & book notes

Vol. 2026

June 07, 2024

Observability for Node.js and Postgres Services with the Grafana LGTM Stack

A practical walkthrough for wiring pino, Grafana Alloy, Loki, and Grafana into a Node.js + Postgres service you can actually debug at 2 a.m.

Introduction

Most Node.js services that talk to Postgres start their observability journey with console.log and a hope that the database never becomes a bottleneck. That works fine until the service is handling real traffic, a query starts timing out intermittently, and nobody can tell whether the problem is in the application, the connection pool, or the database itself. At that point, ad hoc logging stops being useful and the team needs a structured way to answer three questions quickly: what happened, when did it happen, and why did it happen there and not somewhere else.

The Grafana LGTM stack - Loki for logs, Grafana for visualization, Tempo for traces, and Mimir or Prometheus for metrics - was built to answer exactly those questions, and it has become one of the more common self-hosted or Grafana Cloud-backed observability setups for teams that don't want to pay for a full commercial APM platform but still want correlated logs, metrics, and traces. This post walks through a basic, working setup for a Node.js service backed by Postgres, assuming you already have pino for structured logging, Loki for log storage, Grafana Alloy as your collection agent, and Grafana as your dashboard and query layer. The goal is not to cover every possible configuration, but to give you a setup you can run today and extend as your service grows.

Context: Why Node.js + Postgres Needs a Deliberate Observability Strategy

Node.js services that sit in front of Postgres have a particular failure mode that makes observability non-optional rather than nice-to-have. The event loop is single-threaded, and a slow or blocked query doesn't just delay one request - it can back up the connection pool and degrade every other request being handled concurrently. Unlike a thread-per-request model where one slow database call mostly hurts one thread, a Node.js service under connection pool pressure shows up as a systemic slowdown that's hard to diagnose from application logs alone.

This is compounded by the fact that most Node Postgres clients (pg, pg-pool, knex, Prisma's underlying driver) abstract away enough of the connection lifecycle that developers often don't have visibility into pool exhaustion, query queuing time, or long-running transactions until they've already caused an incident. A log line that says "request failed" tells you almost nothing about whether the failure originated in your handler, in query execution, or in waiting for a connection to become available.

The practical answer is to instrument at three levels and correlate them: structured application logs (what the code believed was happening), metrics (aggregate behavior over time - pool utilization, query duration percentiles, error rates), and, where budget and complexity allow, traces (the exact path a single request took, including time spent in Postgres). The Grafana LGTM stack is popular precisely because Grafana lets you pivot between these three signal types from a single pane, using a shared time range and, ideally, shared trace or request identifiers.

The LGTM Stack, Piece by Piece

It's worth being precise about what each component actually does, because the acronym gets used loosely and that leads to confusion about where responsibilities sit. Loki is a log aggregation system designed by Grafana Labs that indexes log metadata (labels) rather than full-text content, which keeps storage costs low compared to something like Elasticsearch, at the cost of less flexible full-text search. Grafana is the visualization and query frontend - it doesn't store data itself but queries Loki, Tempo, Mimir/Prometheus, and dozens of other data sources through a unified UI. Tempo is Grafana's distributed tracing backend, built to store traces cheaply by, similarly to Loki, avoiding a full search index and instead relying on trace IDs for lookups. Mimir (or a plain Prometheus instance, which many smaller setups still use) handles metrics storage and the PromQL query language.

Grafana Alloy is the newer addition to this picture and the one most likely to be unfamiliar if your setup predates 2024. Alloy is Grafana Labs' OpenTelemetry Collector distribution - it replaced the older Grafana Agent and unifies log shipping (previously handled by Promtail), metrics scraping, and OTLP-based trace and metric ingestion into a single binary configured with a component-based configuration language (River, evolving toward Alloy's own syntax). In a setup like yours - pino, Loki, Alloy, and Grafana already in place - Alloy is almost certainly doing double duty: tailing log files or receiving log streams and forwarding them to Loki, and potentially also scraping Prometheus-format metrics endpoints if you've added any. Understanding that Alloy is a collector, not a storage backend, matters when you're debugging pipeline issues, because a "missing logs" problem could be a pino output issue, an Alloy configuration issue, or a Loki ingestion issue, and they require different debugging approaches.

Architecture Walkthrough

A minimal but production-viable setup looks like this: the Node.js service emits structured JSON logs via pino to stdout (or to a file, depending on your container runtime's logging driver). Grafana Alloy runs as a sidecar, daemonset, or standalone agent and is configured to discover and tail those log streams, attach labels (service name, environment, pod name, and so on), and forward them to Loki over HTTP. Separately, if you want metrics, the Node.js process exposes a /metrics endpoint in Prometheus exposition format using a library like prom-client, and Alloy scrapes that endpoint on an interval and forwards the samples to Mimir or Prometheus. Grafana then queries both Loki and your metrics backend, letting you build dashboards that show, for example, Postgres query latency percentiles next to the raw error logs from the same time window.

The reason to route logs through Alloy rather than writing directly to Loki from the application is decoupling. If Loki is temporarily unavailable, Alloy can buffer and retry without your application needing to know anything about the log backend or handle backpressure itself. It also means you can change your log storage backend later without touching application code, since the application's only responsibility is to emit well-structured logs to a local sink.

The one caveat worth flagging up front is that this basic setup, as most teams initially build it, doesn't yet include distributed tracing. Logs and metrics get you most of the way to diagnosing problems in a single service, but if your Node.js service calls other services, or if you want to see exactly how much of a request's total latency was spent inside a specific Postgres query versus in application logic, you eventually want OpenTelemetry traces flowing into Tempo as well. That's addressed later in this post as a natural extension rather than a requirement for day one.

Implementation: Instrumenting the Service

Assuming pino is already configured for structured logging, the first thing worth checking is whether your log lines carry enough context to be useful once they land in Loki. A bare logger.info('query executed') is nearly useless for correlation. Pino supports child loggers and bindings that let you attach a request ID, user ID, or route to every log line within a request's lifecycle, which becomes the join key you'll use when pivoting from a metric spike to the specific logs that explain it.

import pino from 'pino';
import { randomUUID } from 'node:crypto';
import type { Request, Response, NextFunction } from 'express';

export const logger = pino({
  level: process.env.LOG_LEVEL ?? 'info',
  formatters: {
    level(label) {
      return { level: label };
    },
  },
  timestamp: pino.stdTimeFunctions.isoTime,
});

// Attach a request-scoped child logger with a correlation ID.
export function requestLogger(req: Request, res: Response, next: NextFunction) {
  const requestId = (req.headers['x-request-id'] as string) ?? randomUUID();
  req.log = logger.child({ requestId, route: req.path, method: req.method });
  res.setHeader('x-request-id', requestId);
  next();
}

With logging structured this way, Alloy's job is to tail the JSON output and forward it to Loki with labels that make it queryable. A minimal Alloy configuration for log shipping looks roughly like this:

local.file_match "app_logs" {
  path_targets = [{"__path__" = "/var/log/app/*.log"}]
}

loki.source.file "app" {
  targets    = local.file_match.app_logs.targets
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
  external_labels = {
    service = "orders-api",
    env     = env("DEPLOY_ENV"),
  }
}

For metrics, the most immediately valuable thing to expose from a Node.js + Postgres service is connection pool state and query duration, since those are the two signals that most directly explain the failure mode described earlier. Using prom-client alongside the pg driver's pool events gives you this without much overhead:

import { Pool } from 'pg';
import client from 'prom-client';

const pool = new Pool({ connectionString: process.env.DATABASE_URL, max: 20 });

const poolTotal = new client.Gauge({
  name: 'pg_pool_total_count',
  help: 'Total connections in the pool',
});
const poolIdle = new client.Gauge({
  name: 'pg_pool_idle_count',
  help: 'Idle connections in the pool',
});
const poolWaiting = new client.Gauge({
  name: 'pg_pool_waiting_count',
  help: 'Clients waiting for a connection',
});
const queryDuration = new client.Histogram({
  name: 'pg_query_duration_seconds',
  help: 'Postgres query duration in seconds',
  labelNames: ['query_name'],
  buckets: [0.005, 0.01, 0.05, 0.1, 0.5, 1, 2, 5],
});

setInterval(() => {
  poolTotal.set(pool.totalCount);
  poolIdle.set(pool.idleCount);
  poolWaiting.set(pool.waitingCount);
}, 5000);

export async function timedQuery<T>(name: string, text: string, params: unknown[]): Promise<T> {
  const end = queryDuration.startTimer({ query_name: name });
  try {
    const result = await pool.query(text, params);
    return result.rows as T;
  } finally {
    end();
  }
}

Expose those metrics on a dedicated endpoint (app.get('/metrics', ...) returning client.register.metrics()), point an Alloy prometheus.scrape component at it, and you have both logs and metrics flowing without having introduced tracing yet. This is a reasonable stopping point for a first iteration, and it already answers the two most common incident questions: is the pool exhausted, and which query is slow.

Trade-offs and Pitfalls

The most common mistake with this setup is treating Loki like Elasticsearch and querying it with broad, unlabeled searches expecting full-text performance. Loki's design trades search flexibility for cost efficiency by indexing only labels, not log content, so a query that filters on labels first (service, environment, level) and then does a content grep within that narrowed stream will perform far better than one that tries to search across all logs for a string. If your team is used to Kibana-style exploration, this takes some adjustment, and it's worth setting label cardinality guidelines early - high-cardinality labels like user ID or request ID on the Loki side (as opposed to in the log body) will blow up your index and your bill.

The second pitfall is instrumenting metrics without setting cardinality limits on labels, which is a mistake that's easy to make with Postgres query metrics specifically. If you label pg_query_duration_seconds with the raw SQL text or with dynamic values like user IDs, you'll create a new time series for every unique combination, and both Prometheus and Mimir will struggle under that load. Label by a normalized query name or route, not by raw query text or IDs. It's also worth being honest that a basic logs-plus-metrics setup without tracing has a real blind spot: you can see that a request was slow and that a query was slow in roughly the same window, but without a trace ID linking them, you're inferring correlation rather than seeing causation directly. That's an acceptable trade-off for a first observability pass, but it should be treated as a known gap, not an oversight.

Best Practices

Start with consistent labeling conventions before you write a single dashboard. Decide on a small, fixed set of labels - service, environment, and maybe team or region - that every log line, metric, and eventual trace will carry, and enforce it through shared logger and metrics initialization code rather than leaving it to individual developers to remember. This single decision saves more debugging time later than almost any other observability investment, because it's what makes cross-signal correlation in Grafana actually work.

Second, instrument the connection pool and slow query paths before you instrument everything else. It's tempting to add metrics and detailed logging everywhere at once, but the Postgres connection pool is disproportionately likely to be the source of production incidents in a Node.js service, so it deserves priority. Set statement_timeout and idle_in_transaction_session_timeout at the Postgres connection level as a safety net, and make sure timeouts and pool exhaustion errors are logged at a level (warn or error) that's easy to alert on.

Finally, treat Alloy configuration as part of your application's deployment artifact, not as separate infrastructure that lives elsewhere and drifts. When log formats or label schemes change in the application, the Alloy configuration needs to change in lockstep, and keeping both in the same repository or at least the same deployment pipeline avoids the common failure of an application shipping new log fields that Alloy silently drops because it wasn't told about them.

Key Takeaways

Conclusion

A basic Grafana LGTM setup - pino for structured logs, Alloy for collection and forwarding, Loki for storage, and Grafana for querying - gets a Node.js and Postgres service most of the way to being debuggable in production without requiring a commercial APM tool or a large operational investment. The combination of structured, correlated logs and a handful of well-chosen metrics around connection pool health and query duration addresses the failure modes that actually cause incidents in this kind of service.

The natural next step, once this foundation is solid, is adding OpenTelemetry-based tracing into Tempo, which closes the correlation gap between "a request was slow" and "this specific query inside that request was the reason." That's a reasonable second iteration rather than a requirement for launch, and building it on top of consistent labeling conventions established in this basic setup will make the transition significantly less disruptive than trying to introduce all three signal types simultaneously.

References