Observability for Node.js + Postgres in Docker Compose: Adding Prometheus and Mimir to Your Grafana LGTM Stack

Introduction

Most Node.js services start their observability journey with logs. You wire up pino for structured JSON output, point it at Loki through Grafana Alloy, and suddenly you can search and filter application events in Grafana instead of tailing container output. This is a solid foundation, and it is also incomplete. Logs tell you what happened in a specific request or process, but they are poor at answering aggregate questions: how many requests per second is the API handling, what does p99 latency look like over the last hour, how many connections are checked out of the Postgres pool right now. Those questions belong to metrics, and metrics belong to Prometheus-style time series storage.

This article walks through extending a Docker Compose stack that already has Node.js, Postgres, pino, Loki, and Alloy running, by adding Prometheus-compatible metrics collection and Grafana Mimir as the storage backend. The result is a local approximation of the Grafana LGTM stack (Loki, Grafana, Tempo, Mimir) minus tracing, which is enough to give a small service real signal on throughput, latency, error rates, and database pool health, all queryable from a single Grafana instance. The goal is not to build a production-grade platform in a single compose file, but to understand the moving pieces well enough that scaling this pattern to a real cluster later feels familiar rather than foreign.

Why Logs Alone Are Not Observability

It is worth being precise about the gap that metrics fill, because teams sometimes reach for more logging when what they actually need is a counter or a histogram. Logs are discrete events with arbitrary structure. They are excellent for post-hoc investigation: given a request ID or a stack trace, you can reconstruct exactly what a process did. But answering "is the system healthy right now" by scanning logs does not scale, both in the literal cost of ingesting and indexing every line, and in the cognitive cost of eyeballing volume. Prometheus-style metrics are pre-aggregated numeric time series, cheap to store at high cardinality of time but low cardinality of labels, and purpose-built for dashboards, alerting thresholds, and rate-of-change queries like rate(http_requests_total[5m]).

The existing pino -> Loki -> Alloy -> Grafana pipeline in this setup already covers the log axis well. Pino emits structured JSON, a transport ships it to Loki (either directly or via Alloy acting as a log-relay), and Grafana queries Loki with LogQL. What is missing is the metrics axis: request counts, latency histograms, event loop lag, garbage collection pauses, and Postgres connection pool saturation. These are the signals that let you set an alert like "p99 latency over 2s for 5 minutes" without writing log-parsing regex, and they are cheap to collect because the Node.js process already knows these numbers internally.

The third piece worth naming is where Mimir fits relative to Prometheus itself. Prometheus is both a scraper and a time series database; for a single small service you could run Prometheus alone and skip Mimir entirely. Mimir exists because Prometheus's local storage does not scale horizontally or support multi-tenancy well. Since this stack already treats Alloy as the collection layer, and since Alloy speaks the Prometheus remote-write protocol natively, it makes sense to have Alloy scrape metrics and push them into Mimir rather than running a separate long-lived Prometheus server. This mirrors how teams typically evolve from "run Prometheus" to "run Mimir behind a remote-write-capable agent" as they outgrow a single node.

How Alloy Unifies Logs and Metrics Collection

Grafana Alloy is the successor to Grafana Agent, and its configuration model, written in a language Grafana calls "Alloy syntax" (formerly River), is a pipeline of composable components rather than a single monolithic YAML file. Each component has typed inputs and outputs, and you wire them together by referencing one component's exported attributes as another's arguments. This matters for this setup because it means the same Alloy process that already tails or receives pino logs and forwards them to loki.write can also run prometheus.scrape and prometheus.remote_write components side by side, with no separate binary or process to manage.

The scrape model is the standard Prometheus pull model: Alloy is configured with one or more targets (in this case, the Node.js container's /metrics endpoint) and polls them on an interval, by default every 60 seconds though most teams tighten this to 10-15 seconds for application services. The results flow through forward_to, which points at a prometheus.remote_write component configured with Mimir's push endpoint. This is a meaningfully different code path from loki.write, but conceptually identical: collect samples, batch them, ship them to the storage backend over HTTP. Because both pipelines terminate in Alloy, your Docker Compose file gains only one new dependency (Mimir), not two.

Implementation Walkthrough

Assume the existing docker-compose.yml has services for api (the Node.js app), postgres, loki, alloy, and grafana. The first step is instrumenting the Node.js service to expose a /metrics endpoint using prom-client, the de facto standard Prometheus client library for Node.js maintained under the siimon/prom-client repository (the client is also referenced from the official prometheus/client_js GitHub organization). Beyond default process metrics, you want request-level histograms and Postgres pool gauges, since pool exhaustion is one of the most common silent failure modes in Node services talking to Postgres.

// metrics.ts
import client from 'prom-client';
import type { Pool } from 'pg';

export const register = new client.Registry();
client.collectDefaultMetrics({ register });

export const httpRequestDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
  registers: [register],
});

const pgPoolTotal = new client.Gauge({
  name: 'pg_pool_total_count',
  help: 'Total number of clients in the pg pool',
  registers: [register],
});
const pgPoolIdle = new client.Gauge({
  name: 'pg_pool_idle_count',
  help: 'Idle clients in the pg pool',
  registers: [register],
});
const pgPoolWaiting = new client.Gauge({
  name: 'pg_pool_waiting_count',
  help: 'Requests waiting for a client from the pg pool',
  registers: [register],
});

export function observePgPool(pool: Pool): void {
  setInterval(() => {
    pgPoolTotal.set(pool.totalCount);
    pgPoolIdle.set(pool.idleCount);
    pgPoolWaiting.set(pool.waitingCount);
  }, 5000).unref();
}

Wiring this into an Express (or Fastify) app is a small amount of middleware plus one route:

// app.ts
import express from 'express';
import { register, httpRequestDuration, observePgPool } from './metrics';
import { pool } from './db';

const app = express();
observePgPool(pool);

app.use((req, res, next) => {
  const end = httpRequestDuration.startTimer();
  res.on('finish', () => {
    end({ method: req.method, route: req.route?.path ?? req.path, status_code: res.statusCode });
  });
  next();
});

app.get('/metrics', async (_req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

With the application exposing metrics, Alloy needs a scrape target and a remote-write destination alongside its existing log pipeline. The Alloy config file (commonly config.alloy) grows to include:

prometheus.scrape "api" {
  targets = [{ __address__ = "api:3000" }]
  metrics_path = "/metrics"
  scrape_interval = "15s"
  forward_to = [prometheus.remote_write.mimir.receiver]
}

prometheus.remote_write "mimir" {
  endpoint {
    url = "http://mimir:9009/api/v1/push"
  }
}

Mimir itself can run in single-binary mode for local and small-team use, which collapses the distributor, ingester, querier, and store-gateway into one process behind a single -target=all flag. This is the mode used in Grafana's own docker-compose examples for local testing and is more than sufficient for a service-level compose stack:

# docker-compose.yml (excerpt)
services:
  mimir:
    image: grafana/mimir:latest
    command: ["-config.file=/etc/mimir.yaml", "-target=all"]
    ports:
      - "9009:9009"
    volumes:
      - ./mimir.yaml:/etc/mimir.yaml
      - mimir-data:/data

The mimir.yaml file needs, at minimum, a filesystem-backed storage block for local development since Mimir defaults to object storage assumptions in production:

# mimir.yaml
target: all
common:
  storage:
    backend: filesystem
    filesystem:
      dir: /data/blocks
blocks_storage:
  backend: filesystem
  filesystem:
    dir: /data/blocks

Finally, Grafana needs Mimir added as a Prometheus-compatible data source, provisioned the same way Loki likely already is:

# grafana/provisioning/datasources/mimir.yaml
apiVersion: 1
datasources:
  - name: Mimir
    type: prometheus
    access: proxy
    url: http://mimir:9009/prometheus
    isDefault: false

Once wired, http_request_duration_seconds_bucket and pg_pool_waiting_count are queryable in Grafana's Explore view, and you can build a dashboard panel with histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) to see p99 latency trending alongside the log volume panels already backed by Loki.

Trade-offs and Pitfalls

Running Mimir in single-binary, filesystem-backed mode is convenient but it is not representative of how Mimir behaves in production, where it typically runs as microservices against S3-compatible object storage with a separately scaled ingester tier. Anything you learn about Mimir's clustering, replication, or multi-tenant isolation from a local compose file will not transfer; what does transfer is the remote-write protocol and PromQL query surface, since those are identical regardless of deployment topology. Teams sometimes conflate "it works locally" with "this is how it scales," and Mimir's operational documentation is explicit that single-binary mode is intended for evaluation and small-scale use, not multi-tenant production traffic.

A second pitfall is scrape interval and cardinality mismatch between what looks reasonable locally and what is sustainable once you have dozens of services. A 15-second scrape interval on one Node.js process is trivial; the same interval multiplied across 200 services, each emitting default process metrics plus a handful of custom histograms with route and status-code labels, can produce meaningful cardinality growth in Mimir's ingesters. The route label in particular is a common cardinality trap if you accidentally use the raw URL path instead of the matched route template, since /users/123 and /users/456 should collapse to /users/:id, not create two separate series. The example above deliberately uses req.route?.path for this reason.

Third, there is a subtle but important distinction between what Alloy does for logs versus metrics in this architecture. For logs, Alloy is often acting as a receiver or forwarder of already-emitted pino output; for metrics, Alloy is the active scraper, meaning it controls timing and target discovery. If the Node.js container restarts and Docker Compose reassigns it (unlikely with static compose networking, but relevant if you ever move to an orchestrator), static target lists like the one shown above will silently stop working. Production Alloy configurations typically use discovery.docker or discovery.kubernetes components to keep target lists current instead of hardcoding addresses, and it is worth introducing that pattern even in compose if the target list is expected to change.

Best Practices

Keep the metrics surface intentional rather than exhaustive. Default process metrics from prom-client (heap usage, event loop lag, GC pause duration) are cheap and broadly useful, but every custom histogram or counter you add is a maintenance commitment: someone has to decide its buckets, its labels, and eventually its retirement. Start with request duration, error rate (derivable from the status_code label on the duration histogram, so you rarely need a separate error counter), and the Postgres pool gauges shown above. These four signals cover the majority of "is this service healthy" questions for a typical CRUD service, and additional metrics should be added only when a specific incident or SLO gap demonstrates the need.

Treat the Alloy configuration itself as a single source of truth for the collection topology, and keep it under version control alongside the compose file rather than editing it ad hoc inside a running container. Because Alloy's syntax is declarative and component-based, diffs to config.alloy are readable in code review in the same way Terraform diffs are, which makes it easy to see exactly what new scrape target or remote-write destination a change introduces. It is also worth validating the config with alloy fmt and, where available, alloy convert if you are migrating from legacy prometheus.yml-style configuration, so that formatting drift does not obscure real changes.

The 80/20 of This Setup

If you strip away every optional refinement, three things produce almost all of the value: a /metrics endpoint using prom-client's default metrics plus one request-duration histogram, an Alloy prometheus.scrape + prometheus.remote_write pair pointed at Mimir, and one Grafana dashboard panel using histogram_quantile on that histogram. This combination answers "is the service slow or erroring right now" without touching Postgres instrumentation, custom business metrics, or alerting rules at all.

Everything covered after that point, the pool gauges, cardinality hygiene, discovery components, single-binary caveats, matters for running this reliably at scale, but it is refinement on top of a working core, not a prerequisite to get value from the stack. Teams that try to design the "complete" metrics taxonomy before shipping the first histogram usually ship nothing; the practical path is to instrument the request path first, look at what the dashboard actually shows in the first week of real traffic, and add pool or business metrics only in response to a specific question the request histogram cannot answer.

Key Takeaways

Expose a /metrics endpoint with prom-client, starting with default metrics plus one request-duration histogram labeled by method, route template (not raw path), and status code.
Add prometheus.scrape and prometheus.remote_write components to your existing Alloy configuration rather than standing up a separate Prometheus server, since Alloy already handles your log pipeline.
Run Mimir in single-binary mode with filesystem storage for local Docker Compose use, and treat it explicitly as a development approximation, not a production topology.
Instrument the Postgres pg.Pool object directly (totalCount, idleCount, waitingCount) to catch connection pool exhaustion before it manifests as request timeouts.
Provision Mimir as a Grafana data source the same way you likely already provisioned Loki, so metrics and logs live in one Grafana instance without manual setup.

Conclusion

Adding Prometheus-style metrics and Mimir to a Docker Compose stack that already has pino, Loki, Alloy, and Grafana is less about introducing new concepts and more about reusing the collection layer you already trust. Alloy's component model makes metrics scraping a natural sibling to log forwarding rather than a parallel system to operate, and Mimir's single-binary mode gives you a realistic, if not production-scale, place to store and query those metrics using the same PromQL you would use against a full Prometheus deployment.

The broader lesson generalizes beyond this specific stack: observability maturity is usually not about adopting more tools, it is about making the tools you have cover more of the three pillars, logs, metrics, and eventually traces, without fragmenting your query surface. A single Grafana instance querying both Loki and Mimir, fed by a single Alloy process, is a small, well-understood system that scales conceptually to much larger deployments without requiring the team to relearn the mental model.

References

Sources: