paulserban.eu

Writing Edition

Paul Serban

Writing, snippets & book notes

Vol. 2026

Measuring Coaching ROI: Metrics That Actually Matter to CTOs

Moving past 'velocity' to track flow efficiency and developer well-being.

Introduction

Every CTO who has invested in team coaching has faced the same uncomfortable question from the CFO: what did we actually get for that? The honest answer is usually a shrug and a reference to "team morale" or "better collaboration," neither of which maps easily to a line item in a budget. This is not because coaching has no measurable effect - it often has profound ones - but because the engineering industry has defaulted to the wrong set of metrics for decades.

Velocity, measured in story points per sprint, became the de facto language of engineering productivity in the Scrum era. It is convenient, visible, and almost entirely misleading as a coaching ROI signal. Velocity is a planning tool, not a performance indicator. Teams that game it inflate their estimates. Teams that are coached well often see their velocity drop initially as they stop cutting corners and start accounting for technical debt properly. If you are measuring coaching outcomes with velocity, you will frequently conclude that coaching made things worse.

This article is for CTOs and engineering leaders who want to build a defensible, honest measurement framework for coaching ROI. The metrics discussed here are grounded in research - primarily the DORA (DevOps Research and Assessment) program, the SPACE framework developed at Microsoft Research, and Flow Engineering principles from the work of Mik Kersten and others. They are not easy to track, but they are the right ones.

The Problem with Proxy Metrics

The core error in most engineering measurement is confusing activity with outcome. Velocity measures activity. Lines of code measures activity. Number of pull requests merged measures activity. None of these tell you whether the engineering organization is becoming more capable, more sustainable, or more aligned with business goals.

This confusion is not benign. When coaching is evaluated against activity metrics, the incentive structure becomes perverse. Coaches optimize for visible motion rather than systemic change. Teams learn to produce the numbers rather than change the underlying behavior. A team that has genuinely improved its communication patterns, its code review culture, and its approach to technical debt will often produce fewer story points in the short term while setting up far larger capacity gains over the following quarters.

There is also a social dimension to proxy metrics that technical leaders often underestimate. Developers are smart people. They know when they are being measured by a metric that does not reflect the quality of their work. The research literature on this is consistent: surveillance-oriented metrics reduce intrinsic motivation, particularly in knowledge work. A coaching engagement that improves psychological safety and intrinsic motivation can be actively undermined if the measurement framework signals distrust. The choice of metrics is itself a cultural message.

The path forward is not to abandon measurement - it is to measure the right things at the right level of abstraction. Business outcomes sit at the top. Engineering capability indicators sit in the middle. Team health and individual well-being sit at the base. Coaching ROI needs to be visible at all three levels.

The DORA Metrics as a Coaching Baseline

The most empirically validated set of software delivery metrics comes from the DORA research program, published in the annual Accelerate State of DevOps Report and synthesized in the book Accelerate by Nicole Forsgren, Jez Humble, and Gene Kim. The four core DORA metrics - Deployment Frequency, Lead Time for Changes, Change Failure Rate, and Time to Restore Service - have been validated across thousands of organizations as predictors of organizational performance, not just software delivery speed.

These metrics matter for coaching ROI measurement for a specific reason: they are outcome metrics with clear business interpretations. Lead Time for Changes - the time from code commit to running in production - is a direct measure of the feedback loop that development teams operate within. A coaching engagement focused on improving code review practices, reducing handoff delays, or improving test coverage will show up in Lead Time within weeks if the coaching is working. Change Failure Rate - the proportion of deployments that cause a degradation in service - reflects the quality of engineering decisions, the robustness of testing culture, and the clarity of definition-of-done. These are all things that coaching directly addresses.

Establishing a DORA baseline before a coaching engagement begins is essential. Many teams do not have automated measurement pipelines for these metrics. That is acceptable - you can collect them manually from your CI/CD tooling, deployment logs, and incident trackers for an initial baseline. Tools like DORA's own metrics calculator, LinearB, Sleuth, or Cortex can automate this collection once the data sources are wired up. The key is to capture at least eight weeks of pre-coaching baseline data before the intervention begins, and to continue measuring at the same cadence throughout.

One practical caution: DORA metrics reflect the entire delivery system, not just the team being coached. If a coaching engagement focuses on a single team but they are blocked by downstream deployment pipelines controlled by another team, their Lead Time may not improve regardless of how well the coaching lands. Scope your measurement to the variables your coaching actually influences.

Flow Metrics: Seeing the Work System

DORA metrics tell you about delivery outcomes. Flow metrics tell you about the work system itself - how work moves through the pipeline, where it gets stuck, and how much of it is the kind of work the business actually values. The Flow Framework, developed by Mik Kersten in Project to Product, provides a practical vocabulary for this.

Flow Time is the elapsed time from when a work item enters active development to when it is delivered. Unlike Lead Time (which starts from code commit), Flow Time includes everything upstream of the first commit: requirements clarification, design discussion, dependency resolution, and the time items spend sitting in a backlog waiting to be started. This broader view is critical for coaching ROI because a significant portion of waste in engineering organizations lives in these upstream queues. A coaching engagement that improves how teams refine work, write acceptance criteria, or coordinate with product management will show up in Flow Time before it shows up in DORA metrics.

Flow Efficiency is the ratio of active work time to total flow time. In most software teams, this ratio is startlingly low - often in the 15-25% range, meaning work is waiting or blocked for the vast majority of its life in the system. The blocking time is almost always either process (waiting for review, waiting for approval, waiting in a queue) or coordination (unclear requirements, dependencies on other teams, ambiguous ownership). Both of these are precisely the kind of systemic dysfunctions that coaching is designed to address. A coaching engagement that materially improves team autonomy, communication clarity, and cross-functional coordination should produce a measurable improvement in Flow Efficiency within one to two quarters.

// Example: Simple flow efficiency calculator for a work item dataset
interface WorkItem {
  id: string;
  startedAt: Date;
  completedAt: Date;
  activeMinutes: number; // tracked via status transitions in your issue tracker
}

function calculateFlowEfficiency(items: WorkItem[]): number {
  const totalFlowMinutes = items.reduce((sum, item) => {
    const totalMinutes =
      (item.completedAt.getTime() - item.startedAt.getTime()) / 60000;
    return sum + totalMinutes;
  }, 0);

  const totalActiveMinutes = items.reduce(
    (sum, item) => sum + item.activeMinutes,
    0
  );

  return totalActiveMinutes / totalFlowMinutes; // Returns 0-1
}

// Baseline: 0.18 (18% efficiency, typical pre-coaching)
// Target after 2 quarters of coaching: 0.30-0.40

Flow Distribution - the proportion of work that is features versus defects versus technical debt versus risk - is the third critical flow metric for coaching ROI. A team under sustained pressure will shift its flow distribution toward features and away from technical debt and risk reduction. This is not sustainable and eventually shows up as increasing defect rates and decreasing flow velocity. Coaching that helps teams hold the line on technical debt investment should produce a measurable improvement in flow distribution over time, which in turn produces compounding returns in reduced Change Failure Rate and lower maintenance burden.

Developer Experience and the SPACE Framework

The DORA and flow metrics capture the performance of the system. They do not capture whether the humans inside that system are functioning at their cognitive best. This matters for coaching ROI because the most direct effect of good coaching is often on the psychological and relational conditions that enable knowledge workers to do their best thinking.

The SPACE framework, introduced in a 2021 paper by Forsgren et al. in Queue (ACM), organizes developer productivity across five dimensions: Satisfaction and well-being, Performance, Activity, Communication and collaboration, and Efficiency and flow. The key insight of SPACE is that no single dimension is sufficient, and that optimizing for one dimension at the expense of others is the root cause of most productivity measurement failures. A team optimized purely for Activity (high commit rates, high PR throughput) while experiencing declining Satisfaction is a team burning through its capacity.

For coaching ROI measurement, the Satisfaction and Efficiency dimensions are the most directly relevant. Developer satisfaction is typically measured via periodic surveys, the most well-known of which is the Developer Experience (DevEx) survey developed by Abi Noda and colleagues. Key dimensions include: clarity of development environment setup, frequency of flow state interruption, perception of code review quality, and confidence in the deployment pipeline. These are not soft measures - they are leading indicators of attrition risk, productivity capacity, and the psychological safety that determines whether teams surface problems early or hide them.

Measuring developer satisfaction as part of a coaching ROI framework requires a lightweight, consistent survey cadence. Monthly pulse surveys of five to ten questions are more useful than quarterly deep-dive surveys because they allow you to correlate satisfaction changes with specific coaching interventions. When you add a new team ritual, change a process, or address a systemic blocker, you want to see that signal in the next survey cycle, not three months later.

One dimension of developer experience that is particularly sensitive to coaching quality is cognitive load - the degree to which the development environment imposes unnecessary mental overhead. Teams where ownership is unclear, where the codebase lacks consistent patterns, or where deployment requires tribal knowledge are operating under high cognitive load. Coaching that improves code architecture clarity, documentation discipline, and onboarding experience directly reduces cognitive load. This is measurable in onboarding time for new team members - a metric that is easy to track and that proxies well for overall knowledge accessibility in the codebase.

Connecting Metrics to Business Outcomes

The metrics discussed so far - DORA indicators, flow efficiency, developer satisfaction - are meaningful to engineering leaders. They are not, by themselves, persuasive to a CFO. Closing that gap requires a translation layer that connects engineering capability to business value.

The most robust translation is through revenue impact of deployment throughput. For product companies, the ability to ship features more frequently - measured by Deployment Frequency - directly accelerates the pace at which the organization can learn from customers and compound on what works. This is not a hypothetical: the DORA research found that elite engineering organizations deploy 973 times more frequently than low performers. The business implication is that high-performing engineering organizations can run substantially more product experiments per quarter, with a directly proportional effect on their ability to find and optimize winning product directions.

A simpler translation that works for most organizations is the cost of defects. Change Failure Rate has a computable cost: each incident requires engineering time to diagnose and remediate, has customer-facing SLA implications, and carries opportunity cost in the form of feature work displaced. If you track the engineering-hours-per-incident before and after a coaching engagement, and you have a loaded cost per engineering hour, you can compute a direct dollar figure for the reduction in failure rate. A team that reduces its Change Failure Rate from 20% to 8% over two quarters has reduced incident volume by 60%, which at even modest incident resolution costs produces a clear positive ROI against coaching fees.

Retention is the most underweighted factor in coaching ROI calculations. The cost of losing a senior engineer - accounting for recruitment, onboarding, and the ramp time to full productivity - is typically estimated at 0.5-2x annual salary. Coaching engagements that materially improve developer satisfaction and reduce burnout risk have a retention effect that often dwarfs any operational efficiency gains. If a six-month coaching engagement contributes to retaining even one engineer who would otherwise have left, the ROI calculation is typically positive on that factor alone. This requires connecting your developer satisfaction survey trends to your voluntary attrition data - a connection that HR and engineering leadership do not always make explicit but that is straightforward to instrument.

Implementation: Building Your Measurement System

The practical challenge of coaching ROI measurement is that it requires instrumentation that most engineering organizations do not have in place before they start a coaching engagement. The following sequence is designed to be implementable within the first two weeks of any coaching engagement.

Start with your deployment pipeline. Whatever CI/CD tooling you use - GitHub Actions, GitLab CI, Jenkins, Buildkite - will have logs that contain the data you need for DORA metrics. Deployment Frequency and Lead Time can be computed from commit timestamps and deployment event timestamps. If you do not have automated deployments, your deployment runbook or Jira release tickets serve as a proxy. Change Failure Rate requires connecting deployment events to incident or revert events, which may mean joining data from your deployment logs with your incident management system (PagerDuty, OpsGenie, or similar).

# Example: Computing Lead Time for Changes from GitHub data
# Requires: PyGithub, access to your repository's deployment events

from github import Github
from datetime import datetime, timezone
from statistics import median

def compute_lead_time(repo_name: str, token: str, days: int = 30) -> dict:
    g = Github(token)
    repo = g.get_repo(repo_name)

    lead_times = []
    deployments = repo.get_deployments(environment="production")

    for deployment in deployments:
        # Get the commit associated with this deployment
        commit = repo.get_commit(deployment.sha)
        commit_time = commit.commit.author.date

        # Deployment time is when the deployment was created
        deploy_time = deployment.created_at

        if (datetime.now(timezone.utc) - deploy_time).days <= days:
            delta_hours = (deploy_time - commit_time).total_seconds() / 3600
            lead_times.append(delta_hours)

    return {
        "median_lead_time_hours": median(lead_times) if lead_times else None,
        "sample_size": len(lead_times),
        "period_days": days,
    }

Flow metrics require your issue tracker - Jira, Linear, GitHub Issues - to have consistent status transitions and timestamps. Most modern issue trackers log status change events with timestamps; you need to export that data and compute time-in-status for each work item. The key status transitions to capture are: when work enters "In Progress" (Flow Time start), when it is blocked versus active, and when it is deployed or closed (Flow Time end). If your issue tracker does not distinguish blocked from in-progress, add a "Blocked" status - this is a low-friction change that pays dividends in system visibility.

Developer satisfaction measurement requires a survey tool and a consistent cadence. For small teams (under twenty people), a shared Google Form run monthly is sufficient. For larger engineering organizations, purpose-built tools like Pluralsight Flow (which also captures engineering metrics), DX (Developer Experience platform), or even a simple Typeform with a stable question set deliver better participation rates and trend visualization. The survey itself should take no more than three minutes to complete; participation drops sharply beyond that threshold.

Pitfalls and Anti-Patterns

The most common failure mode in coaching ROI measurement is measuring too early and drawing the wrong conclusions. Most coaching engagements produce a J-curve: an initial period where things appear to get worse (teams slow down as they change habits, surface problems that were previously hidden, or invest in technical debt payoff) followed by a more sustained improvement phase. If your CFO checks the Lead Time metric at the eight-week mark of a six-month engagement and sees it has increased, that is not evidence the coaching is failing - it may be evidence it is working. Establish upfront agreements about the measurement horizon before the engagement begins.

A second common pitfall is measuring individuals rather than systems. Coaching ROI is a team-level and organization-level question. Individual commit frequency, individual PR merge rates, or individual story point completion are not coaching outcomes - they are surveillance data. Beyond the ethical and motivational problems with individual-level productivity surveillance, they produce misleading signals: a developer who is doing the hard work of unblocking three colleagues, improving shared documentation, and conducting thorough code reviews will appear "unproductive" in individual activity metrics while generating enormous team-level value.

The third pitfall is treating metric improvement as the goal rather than the symptom. Teams that understand they are being evaluated on Change Failure Rate will start defining incidents more conservatively. Teams evaluated on Deployment Frequency will start deploying smaller changes - which is actually a good engineering practice, so this is a relatively benign form of metric gaming, but it illustrates the general principle. The goal is a more capable, sustainable engineering organization. The metrics are imperfect signals of that goal. Healthy measurement cultures treat the metrics as conversation starters, not verdicts.

Key Takeaways

Five things you can apply immediately to measure coaching ROI more effectively:

  1. Establish a DORA baseline in the first two weeks. Collect eight weeks of pre-coaching data on Lead Time, Deployment Frequency, Change Failure Rate, and Time to Restore. This creates the counterfactual you need for post-engagement comparison.

  2. Add a "Blocked" status to your issue tracker. This single change makes Flow Efficiency computable from data you already have, and it makes invisible blockers visible to the whole team.

  3. Launch a monthly developer satisfaction pulse survey before the coaching begins. Five to eight questions, anonymous, with a stable question set. Trend data is what matters, not any single score.

  4. Translate at least one engineering metric to dollars. Compute the loaded cost of your current Change Failure Rate (incidents * resolution hours * loaded hourly rate). This gives you a floor for the ROI conversation with finance.

  5. Set explicit measurement horizons upfront. Document the expected J-curve and agree with stakeholders that the twelve-week mark, not the four-week mark, is the earliest meaningful signal point.

The 80/20 Insight

If you only change one thing, measure Lead Time for Changes weekly and share it openly with the team. It is the single metric that most directly reflects the health of the feedback loops teams operate within. Coaching that addresses code review culture, deployment pipeline reliability, work item clarity, and team communication will all show up in Lead Time. It is not a perfect metric, but it is the closest thing to a single-number proxy for engineering system health that the field has produced. Teams that see their own Lead Time trend downward develop an empirical relationship with process improvement that compounds over time - they start identifying and removing constraints on their own, which is the real output that good coaching produces.

Conclusion

Measuring coaching ROI in software engineering is not a solved problem, but it is far more tractable than most technical leaders assume. The obstacle is not a lack of data - modern engineering environments generate enormous quantities of it. The obstacle is the persistent habit of measuring activity rather than outcomes, and of looking at individuals rather than systems.

The framework outlined here - DORA metrics for delivery capability, flow metrics for system health, SPACE-informed developer satisfaction surveys for human capital health, and a translation layer to business outcomes - provides a measurement approach that is defensible to finance, meaningful to engineers, and sensitive enough to detect the real effects of coaching over a reasonable time horizon.

The deeper payoff is organizational. Teams that develop a healthy relationship with their own metrics - that use data to identify constraints rather than to assign blame, and that can see the connection between their day-to-day practices and their business outcomes - become self-improving systems. That is what good coaching is ultimately building. The metrics are how you know it is happening.

References