Introduction
The LGTM stack - Loki for logs, Grafana for visualization, Tempo for traces, and Mimir for metrics - has become a common answer to the question of how to run observability without paying Elasticsearch-scale operational costs. Loki in particular built its reputation on a simple premise: index only the metadata (labels), not the full text of every log line, and let object storage handle the heavy lifting. That design choice makes Loki cheap to run and easy to reason about, but it pushes an important decision back onto the platform team - how do logs actually get from a container to Loki in the first place?
This is where the sidecar pattern enters the conversation. In Kubernetes, a sidecar is a second container that runs alongside your application container inside the same pod, sharing its network namespace and, often, its filesystem volumes. When people talk about "running Loki as a sidecar", they usually mean running a log-shipping agent - historically Promtail, increasingly Grafana Alloy - as that second container, dedicated to tailing logs from the application and forwarding them to a central Loki instance. This article walks through what that setup actually looks like, why teams reach for it instead of the more common DaemonSet approach, and when it is genuinely the right call versus when it is unnecessary complexity.
Context: How Logs Get Into Loki
Loki itself does not scrape or tail anything. It is a passive receiver: it exposes a push API (and, for pull-based setups, can work with the OpenTelemetry Collector or other agents) and stores whatever compressed log streams arrive, indexed by a small set of labels like namespace, pod, app, or job. All the work of finding log files, parsing them, attaching labels, and shipping the data over the network is delegated to an agent. Grafana Labs originally built Promtail specifically for this job, and it remains the most common agent paired with Loki, though Grafana has been consolidating its agent lineup into Grafana Alloy, a single distributable that can also handle metrics and traces.
There are two dominant deployment topologies for these agents in Kubernetes, and understanding both is necessary before the sidecar pattern makes sense. The first, and by far the most common, is the DaemonSet model: one agent pod runs per node, mounts the node's /var/log/pods (or the container runtime's log directory) as a hostPath volume, and tails every container's log file on that node. This is efficient - one agent process handles logs for potentially dozens of pods - and it requires zero changes to individual application deployments. The second topology is the sidecar model, where the agent runs inside the same pod as the application it is monitoring, typically reading from a shared volume or directly from stdout via a shared process namespace, and shipping only that pod's logs to Loki.
The reason this distinction matters is that DaemonSets assume your application logs to stdout/stderr, which the container runtime (containerd, CRI-O) already writes to a predictable location on the node's disk. Most cloud-native applications follow this convention, described in the twelve-factor app methodology's treatment of logs as event streams. The sidecar pattern exists precisely for the cases where that assumption breaks down.
Deep Technical Explanation: Why a Sidecar at All
The most common reason to reach for a sidecar is that the application does not write logs to stdout. Plenty of legacy or third-party software - commercial databases, Java applications with hardcoded log4j file appenders, vendor-provided containers you cannot modify - writes exclusively to log files inside the container's filesystem. A DaemonSet agent tailing /var/log/pods on the host cannot see inside that container's writable layer unless the application happens to log to a path that gets mounted externally. The sidecar solves this by sharing an emptyDir volume with the application container: the app writes to /var/log/myapp/app.log inside that shared volume, and the sidecar (Promtail or Alloy) tails the same path from its own container, then ships parsed, labeled entries to Loki.
A second reason is isolation and multi-tenancy. In environments where different teams own different pods on a shared cluster, a per-pod sidecar gives each team direct control over their own log pipeline - their own scrape configuration, their own relabeling rules, their own Loki tenant or endpoint - without needing cluster-admin access to a shared DaemonSet's configuration. This matters in regulated environments where log routing needs to be auditable per workload, or where different applications need genuinely different retention or redaction rules applied at collection time rather than centrally.
A third, more nuanced reason involves multi-line log parsing and per-application processing complexity. Promtail and Alloy support pipeline stages - regex, JSON parsing, multiline stitching for stack traces - but when every application on a node has wildly different log formats, a single shared DaemonSet configuration becomes an unwieldy pile of per-job pipeline stages. A sidecar lets you colocate the parsing logic with the specific application it belongs to, keeping configuration close to the workload it describes. This is the same argument used for sidecar proxies in service mesh architectures: local, per-workload configuration beats a single centralized policy trying to handle every case.
It's worth being precise about what "sidecar" means here versus the broader LGTM stack. Loki itself is never deployed as a sidecar - it is a stateful, horizontally scalable service (in microservices mode: distributor, ingester, querier, compactor, and so on, or a simpler "single binary" / "simple scalable" deployment for smaller installs). What runs as a sidecar is exclusively the collection agent. Conflating the two is a common point of confusion for teams new to the stack, so it is worth stating directly: you are running an agent as a sidecar, not the Loki server itself.
# promtail-sidecar.yaml - a pod with an app container and a Promtail sidecar
# sharing an emptyDir volume for log files the app writes directly to disk.
apiVersion: v1
kind: Pod
metadata:
name: legacy-billing-service
labels:
app: legacy-billing-service
spec:
containers:
- name: billing-app
image: registry.internal/legacy-billing:2.4.1
volumeMounts:
- name: app-logs
mountPath: /var/log/billing
- name: promtail-sidecar
image: grafana/promtail:3.1.0
args:
- -config.file=/etc/promtail/config.yaml
volumeMounts:
- name: app-logs
mountPath: /var/log/billing
readOnly: true
- name: promtail-config
mountPath: /etc/promtail
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
memory: 128Mi
volumes:
- name: app-logs
emptyDir: {}
- name: promtail-config
configMap:
name: billing-promtail-config
# billing-promtail-config ConfigMap contents - scrapes the shared volume,
# stitches multiline Java stack traces, and pushes labeled entries to Loki.
server:
http_listen_port: 9080
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki-gateway.observability.svc.cluster.local:3100/loki/api/v1/push
tenant_id: billing-team
scrape_configs:
- job_name: billing-app-logs
static_configs:
- targets: [localhost]
labels:
app: legacy-billing-service
env: production
__path__: /var/log/billing/*.log
pipeline_stages:
- multiline:
firstline: '^\d{4}-\d{2}-\d{2}'
max_wait_time: 3s
- regex:
expression: '^(?P<timestamp>\S+ \S+) (?P<level>\w+) (?P<message>.*)$'
- labels:
level:
- timestamp:
source: timestamp
format: '2006-01-02 15:04:05'
Implementation Considerations in Practice
Deploying this pattern in a real cluster usually means injecting the sidecar automatically rather than hand-writing it into every pod spec, since maintaining dozens of near-identical sidecar definitions across deployments is a maintenance burden nobody wants. The common approach is a mutating admission webhook or a Helm chart pattern that injects the Promtail or Alloy container based on a pod annotation, similar to how Istio injects its Envoy sidecar. Grafana's own tooling does not ship a first-party injector as polished as Istio's, so most teams either build a small webhook internally or standardize on a Helm subchart that templates the sidecar alongside the main container definition, keeping the emptyDir volume, resource limits, and Promtail config generation consistent across services.
Resource sizing deserves explicit attention because it is easy to get wrong in both directions. A Promtail or Alloy sidecar tailing a single application's logs is lightweight - tens of megabytes of memory and a fraction of a CPU core under normal load - but log bursts (a crash loop producing thousands of stack trace lines per second) can spike memory usage during multiline buffering. Setting a memory limit too low causes the sidecar itself to be OOMKilled, which silently stops log collection for that pod without necessarily affecting the application container. Because Kubernetes restarts containers independently within a pod, a crashed sidecar can go unnoticed unless you are also monitoring the sidecar's own health, which means shipping the sidecar's own logs and metrics somewhere visible - ironically, often to the very same Loki and Mimir instances it feeds.
Migration from Promtail also needs to be part of any 2026-era implementation plan. Grafana Labs put Promtail into long-term support and has been directing new deployments toward Grafana Alloy, which unifies log, metric, and trace collection into a single agent built on the OpenTelemetry Collector's component model. For sidecar deployments specifically, Alloy's advantage is that a single agent binary can simultaneously tail logs for Loki and export traces for Tempo from the same pod, which reduces sidecar count in pods that need both. The configuration language differs (Alloy uses its own River/HCL-like syntax rather than Promtail's YAML), so teams with an established Promtail sidecar fleet should treat the migration as a deliberate project rather than a drop-in replacement, testing pipeline stage equivalence carefully since regex and multiline behavior are not always identical between the two.
Trade-offs and Common Pitfalls
The sidecar pattern's central cost is resource multiplication. Every pod that gets a sidecar pays for an additional container's memory and CPU reservation, and at fleet scale - hundreds or thousands of pods - this adds up to a meaningfully larger total resource footprint than a handful of DaemonSet pods per node handling the same log volume. A DaemonSet agent amortizes its fixed overhead (Go runtime, TCP connections to Loki, buffering logic) across every pod on the node; a sidecar pays that fixed cost per pod. Teams sometimes underestimate this until a cluster-wide resource audit reveals that logging sidecars account for a surprising fraction of total requested memory across the fleet.
There is also an operational surface-area cost. Each sidecar is a container that can crash, get stuck, fall behind on its Loki push queue, or misconfigure its pipeline stages - and because it lives inside application pod specs rather than a single centrally managed DaemonSet, a configuration bug can be scattered across many slightly different YAML files rather than living in one place. This is the classic sidecar trade-off familiar from service mesh adoption: pushing configuration down to the workload level increases flexibility but decreases the ease of making a single global change. A label schema change or a Loki endpoint migration that would be a one-line edit to a DaemonSet's ConfigMap becomes a fleet-wide rollout across every sidecar-injected deployment.
A subtler pitfall is label cardinality. Loki's entire performance model depends on keeping the set of indexed labels small and low-cardinality - this is explicitly called out in Grafana's own Loki documentation as the single most common cause of poor Loki performance. Sidecar deployments, because they are configured per-application, make it easy for individual teams to add pod-specific or even request-specific labels (a unique request ID, a pod name with a random suffix) without central review, since there's no shared configuration file where such mistakes would be visible to a platform team doing a review pass. This can quietly degrade query performance across the shared Loki instance even though the mistake originated in one team's isolated sidecar config. Enforcing a labeling standard via policy (OPA/Gatekeeper, Kyverno) or a shared Helm chart that only exposes a safe subset of templatable labels is the practical mitigation.
Finally, sidecars introduce a lifecycle coupling problem: when a pod terminates, both containers are typically sent SIGTERM roughly simultaneously, which risks losing the final burst of logs written just before shutdown if the sidecar isn't given a moment to flush and drain its send queue. Kubernetes does not guarantee sidecar shutdown ordering by default (though native sidecar containers, stabilized in Kubernetes 1.29 via restartPolicy: Always init containers, improve startup and shutdown ordering guarantees). Teams running sidecar-based logging in mixed-version clusters need to verify their Kubernetes version actually supports native sidecar semantics before relying on them.
Best Practices for Running Loki Agents as Sidecars
Default to the DaemonSet model and reserve sidecars for genuine exceptions. This is the single most important piece of guidance in this entire topic: most Kubernetes-native applications that log to stdout have no need for a sidecar at all, and reaching for one out of habit or unfamiliarity with the alternative simply adds cost and operational surface area. Reserve the sidecar pattern for applications that cannot log to stdout, that need genuinely isolated per-team log pipelines, or that require processing complex enough to warrant dedicated configuration.
When a sidecar is justified, standardize its deployment through a shared Helm chart or Kustomize base rather than letting each team hand-roll their own pod spec. Centralizing the sidecar container definition, resource limits, and the safe subset of configurable fields (log paths, allowed labels, Loki tenant) prevents the configuration drift and cardinality problems described above while still giving teams the isolation benefit that motivated the sidecar in the first place.
Set explicit, tested resource limits on the sidecar container and monitor its own health independently of the application container. Treat the Promtail or Alloy sidecar as a first-class workload with its own alerting - specifically, alert on the agent's send queue depth and its own error logs (both agents expose Prometheus metrics for this), since a sidecar silently falling behind on delivery is worse than an obvious crash, because it produces gaps in your logs that you may not notice until you need them during an incident.
Plan your Promtail-to-Alloy migration deliberately rather than reactively. Grafana Labs' own migration guidance recommends validating pipeline stage equivalence in a non-production environment before cutting over, given the syntax differences between Promtail's YAML and Alloy's configuration language. Doing this migration calmly, ahead of any forced deprecation timeline, is preferable to doing it under pressure.
Key Takeaways
- Use a DaemonSet by default. It is cheaper, simpler, and sufficient for any application that logs to stdout/stderr - which covers the overwhelming majority of cloud-native workloads.
- Reach for a sidecar only for specific reasons: applications that write logs exclusively to files inside the container, per-team pipeline isolation requirements, or per-application parsing complexity that would clutter a shared DaemonSet config.
- Centralize sidecar configuration through a shared chart or template rather than letting it be hand-written per deployment, to avoid configuration drift and label cardinality mistakes.
- Monitor the agent, not just the application. A sidecar that silently falls behind or crashes produces a logging blind spot that is easy to miss until an incident makes it obvious.
- Treat Loki (the server) and the collection agent (Promtail/Alloy) as separate architectural decisions. Loki is never a sidecar; only the log-shipping agent is, and conflating the two leads to confused capacity planning.
Conclusion
The sidecar pattern for Loki-bound log collection is not a default architecture - it is a targeted solution to a specific set of problems: applications that cannot log to stdout, teams that need isolated pipeline control, or per-application parsing complexity that overwhelms a shared configuration. For the majority of Kubernetes workloads that already follow twelve-factor logging conventions, a DaemonSet running Promtail or Grafana Alloy remains the simpler, cheaper, and more maintainable choice. Understanding both models, and the specific trade-offs each makes around resource cost, configuration centralization, and failure isolation, is what allows a platform team to make this decision deliberately rather than by default or by copying whatever pattern a blog post recommended without context.
As Grafana continues consolidating its agent ecosystem around Alloy, the practical mechanics of this decision will keep shifting, but the underlying architectural question - does this workload's logging needs justify per-pod isolation, or does centralized node-level collection suffice - will remain the right first question to ask.
References
- Grafana Labs - Loki Documentation: https://grafana.com/docs/loki/latest/
- Grafana Labs - Promtail Documentation: https://grafana.com/docs/loki/latest/send-data/promtail/
- Grafana Labs - Grafana Alloy Documentation: https://grafana.com/docs/alloy/latest/
- Grafana Labs - "Migrating from Promtail to Alloy": https://grafana.com/docs/alloy/latest/tasks/migrate/from-promtail/
- Grafana Labs - Loki Best Practices (label cardinality guidance): https://grafana.com/docs/loki/latest/best-practices/
- Kubernetes Documentation - Sidecar Containers: https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/
- Kubernetes Documentation - Logging Architecture: https://kubernetes.io/docs/concepts/cluster-administration/logging/
- The Twelve-Factor App - Logs: https://12factor.net/logs
- OpenTelemetry Collector Documentation: https://opentelemetry.io/docs/collector/
- Kubernetes Enhancement Proposal - Sidecar Containers (KEP-753, native sidecars stabilized in 1.29): https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/753-sidecar-containers