paulserban.eu

Writing Edition

Paul Serban

Writing, snippets & book notes

Vol. 2026

DevOps Culture: The Importance of Blameless Postmortems

Why psychological safety, not root cause analysis, is the real engine of operational reliability

Introduction

Every sufficiently complex system eventually fails. Databases go down under unexpected load, deployments introduce regressions that slip past test suites, and cascading failures unravel what looked like bulletproof architecture on paper. The question is never whether incidents will happen - it is what an engineering organization learns from them when they do.

The traditional response to failure in many organizations is to find the person who made the wrong change, escalate the problem to their manager, and treat the incident as a personnel issue. This approach feels intuitive. Someone made a mistake; someone should be accountable. But this reasoning conflates accountability with blame, and in doing so, it destroys the very conditions that allow teams to improve over time. Engineers who fear punishment stop reporting near-misses. They avoid risky-but-important changes. They learn to cover tracks rather than examine root causes.

Blameless postmortems emerged as a deliberate rejection of this pattern. Pioneered at scale by companies like Google and Etsy, the practice reframes failure as a systemic event rather than a personal one, and turns incident reviews into structured learning exercises. This article examines why blameless postmortems work, how to run them effectively, where organizations commonly go wrong, and what separates a high-quality postmortem culture from a performative one.

The Problem with Blame-Driven Incident Reviews

Before understanding what blameless postmortems solve, it helps to understand what blame-driven reviews produce. When an incident review's implicit goal is to identify who is at fault, the dynamics in the room shift in predictable and damaging ways.

Engineers under scrutiny focus on self-protection rather than honest reconstruction. They present timelines selectively, avoid mentioning their own confusion during the incident, and minimize the significance of decisions they made under pressure. Managers who attend to "keep an eye on things" inadvertently raise the stakes for everyone. The result is an incident review that produces a clean narrative with a single identifiable cause - usually a human one - and a corrective action that amounts to "be more careful next time." This tells the organization almost nothing useful.

The deeper problem is that this approach misunderstands how failures actually occur in complex systems. Sidney Dekker's work in the field of human factors and safety engineering, particularly in The Field Guide to Understanding 'Human Error' (2006), established that so-called human error is almost always the effect of systemic conditions, not the cause of them. Engineers make decisions with incomplete information, under time pressure, using tools that obscure the true state of the system. When a deployment fails because an engineer didn't notice a subtle configuration drift, the proximate cause is the missed configuration - but the real causes are a deployment pipeline without drift detection, a review process that couldn't catch it, and possibly an alert system that didn't surface the anomaly until it was too late.

Treating the engineer as the root cause removes the incentive to fix any of those systemic problems. It also removes the engineer's incentive to be honest in the next incident review, because honesty now has a known cost.

What Blameless Postmortems Actually Are

The term "blameless postmortem" is widely used but often imprecisely understood. A blameless postmortem is not a meeting where blame is forbidden by social convention, or where everyone agrees to be polite. It is a structured process designed to produce an accurate, honest account of how an incident happened, with the explicit organizational guarantee that participating honestly will not result in punishment for the individuals involved.

That organizational guarantee is the load-bearing element. Without it, blamelessness is theater. With it, engineers can reconstruct timelines that include moments of confusion, false assumptions, and decisions that seemed reasonable at the time - which is precisely the information that reveals what the system actually did versus what engineers believed it was doing.

Google's Site Reliability Engineering (SRE) book, freely available at sre.google, describes postmortems as a core SRE practice and specifies several criteria for what makes one effective: a complete timeline, identification of contributing causes (plural), a set of action items with clear owners, and a section explicitly listing what went well. That last item - what went well - is often omitted in organizations new to the practice, but it matters. It signals that the review is not a prosecution and gives the team credit for the detection and mitigation work that limited the blast radius.

Etsy's contribution to formalizing blameless postmortems, documented by John Allspaw in his 2012 blog post "Blameless PostMortems and a Just Culture," introduced another critical concept: separating the second story from the first story. The first story is the simplified narrative constructed after the fact - "an engineer deployed a bad config." The second story is the actual sequence of events as experienced by the people involved - what they knew, when they knew it, what the monitoring showed, what the runbooks said, what happened when the first fix didn't work. Blameless postmortems are explicitly in the business of recovering the second story.

The Mechanics: How to Run an Effective Postmortem

Running a postmortem well is a skill that takes practice. The process has several distinct phases, each of which can go wrong in its own way.

Triggering Criteria

Not every incident needs a full postmortem, but teams should define explicit triggering criteria rather than leaving it to judgment. Common triggers include: any customer-visible outage lasting more than a threshold duration (e.g., five minutes), any incident that required an on-call escalation, any data loss event regardless of scale, and any security incident. The threshold should err toward inclusion; a postmortem that turns out to be straightforward costs little, while a skipped postmortem on a complex incident costs the next time the same failure mode recurs.

Timeline Construction

The most important artifact a postmortem produces is an accurate, granular timeline of the incident. This should be constructed before the postmortem meeting itself, using log data, monitoring dashboards, alert history, chat logs, and on-call notes - not from memory. Memory is reconstructive and converges toward narratives that make sense retrospectively. Primary sources preserve the genuine sequence, including the false starts, the monitoring blind spots, and the moments where engineers were working with incorrect mental models.

The timeline should include both system events (alerts fired, deploys kicked off, circuit breakers tripped) and human actions (who ran which command, when the incident commander made the call to rollback, when external teams were looped in). This dual record reveals the interaction between automated systems and human judgment that characterizes most real incidents.

The Meeting Itself

The postmortem meeting should be moderated by someone who was not deeply involved in the incident response - ideally a designated facilitator whose role is to keep the conversation on the second story and redirect any discussion that starts assigning personal fault. The facilitator's most important tool is the question: "What did you know at that point, and what were you trying to accomplish?" This reframes decisions in their original context rather than judging them with hindsight.

Attendees should include the on-call engineers who responded, the team owning the affected systems, and any stakeholders from dependent teams. Management attendance is appropriate only if the organization has genuinely internalized blamelessness; otherwise, their presence shifts the room's dynamics in ways that are hard to counteract. Larger organizations sometimes run two sessions: a detailed technical postmortem for the engineering team, followed by a summarized executive briefing with the action items.

Action Items

The postmortem's value is ultimately realized in its action items. These should be specific, assignable, and tied directly to the contributing causes identified in the timeline. An action item like "improve monitoring" is not actionable; "add an alert on the p99 latency of the payments API with a threshold of 500ms" is. Each action item should have a named owner, a target date, and a priority.

A useful heuristic from the SRE practice is to distinguish between immediate mitigations (things done during the incident to stop the bleeding), short-term fixes (things to be done within days or a week), and long-term systemic improvements (architectural changes, tooling investments, process changes). Postmortems that only produce short-term fixes without systemic improvements are doing half the job.

Practical Implementation: Templates and Tooling

Many organizations benefit from a standardized postmortem template that enforces the structure without constraining the content. Below is a reference template that reflects established SRE practices.

# Postmortem: [Incident Title]

**Date:** YYYY-MM-DD  
**Severity:** SEV-1 / SEV-2 / SEV-3  
**Duration:** HH:MM  
**Incident Commander:** [Name]  
**Author(s):** [Names]  
**Status:** Draft / In Review / Final

---

## Impact Summary

Brief description of what was affected and for how long.

- Affected services:
- Customer impact:
- Revenue/SLO impact (if applicable):

---

## Timeline

| Time (UTC) | Event                            |
| ---------- | -------------------------------- |
| HH:MM      | Anomaly first visible in metrics |
| HH:MM      | Alert fired / on-call paged      |
| HH:MM      | Investigation began              |
| HH:MM      | Root cause hypothesized          |
| HH:MM      | Mitigation applied               |
| HH:MM      | Recovery confirmed               |
| HH:MM      | Incident closed                  |

---

## Contributing Causes

List all contributing factors. Avoid single root cause framing.

1. ...
2. ...
3. ...

---

## What Went Well

- Detection was fast because ...
- The rollback procedure worked as expected ...
- Cross-team communication was effective ...

---

## Action Items

| Item                                        | Owner     | Priority | Due Date   | Status |
| ------------------------------------------- | --------- | -------- | ---------- | ------ |
| Add p99 latency alert on payments API       | @engineer | P1       | YYYY-MM-DD | Open   |
| Investigate config drift detection in CI/CD | @team     | P2       | YYYY-MM-DD | Open   |

---

## Lessons Learned

Narrative summary of what the team now understands about the system that it did not before.

For teams integrating postmortem workflows into their existing toolchain, several purpose-built tools exist. PagerDuty offers built-in postmortem functionality tied to its incident timeline. Blameless (the company) offers a dedicated platform. For teams already using Confluence or Notion, a structured template in those tools works equally well - the tooling matters less than the discipline of filling it out consistently and tracking action items to completion.

The real integration challenge is not the document itself but the action item lifecycle. Action items written in a postmortem and never revisited are a common failure mode. The most reliable fix is to treat postmortem action items as first-class work items in the team's project tracker (GitHub Issues, Jira, Linear), with the same prioritization and review cadence as feature work. Some teams designate a "postmortem champion" rotation responsible for reviewing action item completion before each sprint.

# Example: GitHub Issues integration for postmortem action items
# Run after postmortem meeting to create tracked issues automatically

import requests
import json
from dataclasses import dataclass
from typing import List

@dataclass
class ActionItem:
    title: str
    owner: str
    priority: str  # P1, P2, P3
    due_date: str
    postmortem_url: str

def create_github_issue(action_item: ActionItem, repo: str, token: str) -> dict:
    """
    Creates a GitHub issue from a postmortem action item.
    Labels map to priority; body links back to the postmortem doc.
    """
    priority_labels = {"P1": "priority:critical", "P2": "priority:high", "P3": "priority:medium"}

    payload = {
        "title": f"[Postmortem Action] {action_item.title}",
        "body": (
            f"**Owner:** @{action_item.owner}\n"
            f"**Due:** {action_item.due_date}\n"
            f"**Source postmortem:** {action_item.postmortem_url}\n\n"
            "---\n"
            "_This issue was auto-created from a postmortem action item. "
            "Close it only when the underlying fix is verified in production._"
        ),
        "labels": [priority_labels.get(action_item.priority, "postmortem"), "postmortem"],
        "assignees": [action_item.owner],
    }

    response = requests.post(
        f"https://api.github.com/repos/{repo}/issues",
        headers={
            "Authorization": f"Bearer {token}",
            "Accept": "application/vnd.github+json",
        },
        data=json.dumps(payload),
    )
    response.raise_for_status()
    return response.json()


def sync_postmortem_actions(action_items: List[ActionItem], repo: str, token: str) -> None:
    for item in action_items:
        issue = create_github_issue(item, repo, token)
        print(f"Created issue #{issue['number']}: {issue['html_url']}")

Trade-offs and Pitfalls

Blameless postmortems are not a silver bullet, and organizations frequently encounter several predictable failure modes when implementing them.

Blamelessness without systemic change is frustrating. If engineers participate honestly in postmortems and watch the same categories of issues recur because action items are deprioritized, they will stop investing effort in honest participation. Blameless culture requires that the organization actually act on what postmortems reveal. This is a leadership problem more than an engineering one.

The "just culture" boundary is genuinely difficult. John Allspaw's formulation of blameless postmortems references Sidney Dekker's concept of a "just culture" - a culture that distinguishes between acceptable risk-taking and reckless behavior. Pure blamelessness can create a perverse dynamic where genuinely negligent behavior has no consequences. Most mature organizations converge on something like Dekker's substitution test: would another engineer with similar training and context have made the same decision? If yes, the failure is systemic. If no, a personnel conversation may still be warranted - but that conversation happens separately and does not contaminate the postmortem.

Postmortems can become bureaucratic overhead. Organizations that require postmortems for minor incidents, use templates that demand excessive documentation for every section, or run postmortems in large all-hands formats rather than focused small-group sessions often find that engineers begin treating them as compliance exercises. The format should scale with the incident's severity and complexity. A SEV-3 incident that lasted 10 minutes and was handled by two engineers does not need the same postmortem depth as a 4-hour multi-service outage.

Psychological safety cannot be mandated. Leadership can declare blamelessness, but engineers will calibrate their honesty against observed outcomes - not stated policies. If an engineer is honest in a postmortem and then receives negative performance feedback for the actions they described, the policy is dead regardless of what it says on paper. Building genuine psychological safety requires consistent behavior from engineering managers over time, not a policy document.

Best Practices

The following practices are drawn from the SRE book, Allspaw's writing, and the accumulated operational knowledge of teams that have run blameless postmortems at scale.

Write the postmortem within 48-72 hours of the incident. Memory degrades quickly, and the engineers who responded will shift their mental context back to feature work within days. The timeline reconstruction is harder, the details are fuzzier, and the action items are vaguer the longer you wait. Treat the postmortem as part of incident response, not a separate activity that follows it. Name contributing causes in the plural. Complex systems fail through chains of conditions, not single root causes. Templates that ask for "the root cause" encourage teams to stop digging once they find a plausible candidate. Asking for contributing causes (and explicitly expecting multiple) produces deeper analysis and more systemic action items. Publish postmortems broadly. Google publishes postmortems internally to all engineers, not just the teams involved. This builds an institutional memory of failure modes across the organization. Engineers working on new systems can search past postmortems and discover that the problem they're designing around has already been encountered - and either solved or not. Postmortems that stay within the team are only half as valuable. Track the completion rate of action items as a metric. If leadership cares about postmortem quality, it will measure something about postmortems. The most useful metric is not the number of postmortems written but the percentage of action items completed within their target window. An organization completing 80% of postmortem action items on time has a fundamentally different reliability posture than one completing 20%. Hold a quarterly postmortem review. Reviewing postmortems in aggregate on a periodic basis reveals patterns that individual postmortems cannot. If six postmortems in a quarter all have an action item related to observability gaps, that is a signal to invest in observability infrastructure - a conclusion invisible in any single postmortem but obvious in the aggregate.

Analogies and Mental Models

One useful mental model for blameless postmortems comes from aviation. Commercial aviation has one of the strongest safety records of any complex industry, and it got there largely through a systemic approach to incident investigation pioneered by organizations like NASA's Aviation Safety Reporting System (ASRS), which allows pilots to submit confidential reports about safety-relevant incidents without fear of punishment. The result is a rich dataset of near-misses that has driven systemic improvements across the entire industry. The key insight is that near-misses contain the same causal information as actual accidents - and there are far more of them. Aviation learned to treat them as the primary source of safety intelligence.

Another useful framing is the distinction between first-order and second-order effects of blame. The first-order effect is obvious: the blamed engineer feels bad, and that feels like accountability. The second-order effect is what actually matters: engineers across the organization update their behavior to minimize their exposure to blame. They document less, report less, avoid the changes that could cause incidents, and optimize for not being associated with failures rather than for building reliable systems. The organization becomes measurably less reliable as a direct result of the accountability culture it thought was making things more reliable.

80/20 Insight

If you take only one structural change from this article, make it this: decouple the postmortem from the performance review process, institutionally and visibly. This single change does more to unlock honest postmortem participation than any other intervention. It requires explicit communication from engineering managers - not just a policy statement, but individual conversations with engineers confirming that what they say in a postmortem cannot appear in a performance review. Once engineers believe this to be true based on observed behavior, the quality of second-story reconstruction improves dramatically, and the action items that emerge become genuinely systemic rather than cosmetic.

The second highest-leverage practice is tracking action item completion. Most of the value of blameless postmortems is destroyed by action items that are written down and never revisited. Making action item completion a visible, measured team metric creates the organizational pull that ensures postmortems translate into actual reliability improvements.

Key Takeaways

Five things you can apply immediately:

  1. Define your triggering criteria - write down what kinds of incidents require a postmortem and share that definition with your team. Remove ambiguity about when one is expected.
  2. Start with a template - adapt the template provided in this article to fit your organization's context. The specific sections matter less than the discipline of filling them out consistently.
  3. Create GitHub/Jira issues for every action item - don't let action items live only in the postmortem document. Put them in your team's primary project tracker with an owner and due date.
  4. Publish postmortems to the broader engineering org - even a read-only Confluence page or internal wiki entry builds institutional memory and normalizes the practice.
  5. Track action item completion rate - pick a time window (e.g., 30 days) and measure what percentage of action items from that window are closed. Review this metric with your team monthly.

Conclusion

Blameless postmortems are not primarily a documentation practice or a meeting format. They are a commitment to a specific theory of how organizations learn: that honest reconstruction of failure is more valuable than punishment of failure, and that the conditions that allow failures to happen are almost always systemic rather than personal.

The evidence for this approach is well-established in both software engineering and adjacent fields like aviation and nuclear power, which have spent decades building safety cultures grounded in the same principles. Google's SRE practice has made postmortems a foundational discipline for managing systems at a scale where incidents are inevitable. The organizations that have adopted these practices seriously - not just in name but in the behavioral norms and leadership commitments that make them real - have demonstrably better reliability outcomes than those that haven't.

The hardest part is not the process. Templates are easy to find and adapt. The hard part is the cultural shift: convincing engineers through consistent observed behavior that honesty in an incident review will not harm them, and convincing leadership that the value of that honesty outweighs the emotional appeal of having someone to hold accountable when things go wrong. That shift happens slowly, through repeated incidents where the blameless process produced systemic improvements that a blame-driven process would have missed, and through the gradual accumulation of trust between engineers and their organizations.

It is worth building.

References