paulserban.eu

Writing Edition

Paul Serban

Writing, snippets & book notes

Vol. 2026

10 Buildkite Pipeline Optimizations That Cut Build Times by 60%

Real-world techniques used by engineering teams at scale to eliminate bottlenecks and ship faster

Introduction

Every minute a developer waits on CI is a minute pulled out of deep work. At scale - dozens of engineers, hundreds of daily commits - even a modest 10-minute pipeline compounds into thousands of lost engineering hours per month. The compounding doesn't stop at productivity: slow pipelines erode confidence in the feedback loop, encourage batching commits to "save time," and often become the invisible ceiling on a team's deployment frequency.

Buildkite occupies a unique architectural position in the CI/CD landscape. Unlike fully managed SaaS CI platforms, it separates the orchestration layer (Buildkite's cloud) from the execution layer (your own agents). This gives engineering teams unusual leverage: you control the hardware, the OS, the caching infrastructure, and the degree of parallelism. That leverage is powerful, but it also means teams that haven't deliberately optimized their pipelines are often leaving dramatic performance gains on the table.

This guide covers ten concrete optimizations drawn from patterns common in mature Buildkite deployments. None of them require exotic infrastructure. Most can be implemented incrementally in an afternoon. Together, they can realistically reduce a 15-minute pipeline to under 6 minutes - without sacrificing reliability or correctness.

The Real Cost of Slow Pipelines

Before optimizing, it's worth understanding what a slow pipeline actually costs. There is a direct productivity cost: every engineer who pushes a branch and waits 20 minutes before getting signal is context-switching away from the work they just finished. Research on software developer productivity (summarized in works like Accelerate by Forsgren, Humble, and Kim) consistently identifies CI feedback time as a key driver of deployment frequency and change failure rates. Teams with fast pipelines deploy more often and with higher confidence.

There is also a subtler cost: slow pipelines push engineers toward behaviors that make things worse. Fewer commits per day means more code changes bundled into each push. More changes per push means slower, less parallelizable test runs and harder-to-diagnose failures. This feedback loop is self-reinforcing until something breaks it - typically a deliberate investment in pipeline speed.

The Buildkite architecture offers a genuine escape. Because you own the agents, you can right-size them for each job type, spread work across many machines simultaneously, and cache aggressively at every layer. The ten optimizations below are a structured path through that escape.

Optimization 1: Parallelize Steps Aggressively

The single highest-leverage optimization in almost every pipeline is converting sequential steps into parallel ones. Buildkite's pipeline DSL makes this straightforward: any steps not connected by a depends_on relationship run in parallel by default when agents are available.

The key insight is to draw a dependency graph of your pipeline steps rather than defaulting to top-to-bottom ordering. A common antipattern is running lint, type-check, unit tests, and integration tests in sequence when only integration tests actually need the build artifact. If lint and type-check can run directly from source, they have no dependency on the build step and can run in parallel with it.

# BEFORE: Sequential steps - each waits for the previous
steps:
  - label: ":hammer: Build"
    command: "npm run build"
  - label: ":mag: Lint"
    command: "npm run lint"
  - label: ":white_check_mark: Unit Tests"
    command: "npm test"
  - label: ":rocket: Integration Tests"
    command: "npm run test:integration"
    depends_on: "build"

# AFTER: Parallel where possible, sequential only where necessary
steps:
  - label: ":hammer: Build"
    key: "build"
    command: "npm run build"

  - label: ":mag: Lint"
    command: "npm run lint"
    # No depends_on - runs in parallel with build

  - label: ":white_check_mark: Unit Tests"
    command: "npm test"
    # No depends_on - runs in parallel with build

  - label: ":rocket: Integration Tests"
    command: "npm run test:integration"
    depends_on: "build" # Only this step needs the build artifact

The wall-clock time for a pipeline is determined by the longest sequential chain, not the total work. Eliminating unnecessary sequential dependencies collapses that chain.

Optimization 2: Use Parallelism for Test Splitting

Parallelizing steps is one dimension of concurrency. The second dimension is parallelizing within a step - running your test suite across multiple agents simultaneously. Buildkite's parallelism key enables this with minimal configuration.

The challenge is splitting tests intelligently. Naive splitting by file count produces uneven shards if some test files are much slower than others. Buildkite's Test Analytics (formerly Test Collector) tracks per-test timing history and can drive time-based splitting. Alternatively, tools like jest --shard (for JavaScript) or pytest-split (for Python) use historical timing data stored in CI artifacts to produce balanced shards.

steps:
  - label: ":test_tube: Tests (shard %n of %c)"
    command: "pytest --splits 8 --group $$BUILDKITE_PARALLEL_JOB"
    parallelism: 8
    agents:
      queue: "test-runners"
    plugins:
      - artifacts#v1.9.0:
          download: ".test_durations"
          upload: ".test_durations"

Eight agents running parallel shards of a test suite that previously ran on one agent can theoretically reduce test time by 8x. In practice, with reasonable splitting and fast agents, 4-6x improvements are common. The .test_durations artifact carries timing data forward so each subsequent run produces better-balanced shards.

Optimization 3: Layer Your Caching Strategy

Dependency installation is one of the most reliably expensive parts of any pipeline. A Node.js project with 500 dependencies, a Python project with a deep requirements tree, or a Go module cache - all of these represent work that is identical on every run unless the dependency manifest has changed. Caching this work is not merely a nice-to-have; it is essential for fast pipelines.

Buildkite does not provide a managed cache layer out of the box (unlike some SaaS CI platforms), but the cache plugin and the S3/GCS artifact plugins give you the building blocks. The architecture that performs best in practice is a layered cache: a fast local cache on the agent disk for same-agent reuse, and a remote cache (S3, GCS, or a dedicated cache server) for cross-agent reuse.

steps:
  - label: ":package: Install Dependencies"
    command: |
      # Restore cache if available
      cache-restore node_modules $CACHE_KEY
      # Install only if cache miss
      if [ ! -d node_modules ]; then
        npm ci
        cache-save node_modules $CACHE_KEY
      fi
    env:
      CACHE_KEY: "node-modules-v1-$$(sha256sum package-lock.json | cut -d' ' -f1)"

The cache key strategy matters as much as the cache infrastructure. A key scoped to the exact lockfile hash means: hit on identical dependencies, miss on any change. Prefix the key with a version string so you can invalidate all caches deliberately when needed (e.g., after an OS upgrade on agents). Also consider caching build outputs themselves - compiled binaries, webpack output, generated protobuf code - keyed on the source files that produce them.

A well-layered cache strategy typically eliminates 60-80% of dependency installation time on cache hits, which on a project with a 3-minute npm ci translates directly to a 3-minute reduction in pipeline wall time.

Optimization 4: Right-Size and Specialize Your Agent Pools

One of Buildkite's most underused capabilities is its agent queue system. Most teams start with a single agent pool and route everything there. This means a 4-CPU agent that's perfectly sized for lint runs is also handling memory-intensive integration tests that would run twice as fast on a 16-CPU agent with 32 GB of RAM.

Agent specialization pays off quickly. Define queues by job characteristics: a lint queue with cheap, small agents; a unit-test queue with medium agents; an integration queue with larger, network-attached agents; a build queue with agents that have fast NVMe local storage for build caches. Each step in your pipeline specifies its queue, and Buildkite routes accordingly.

steps:
  - label: ":mag: Lint"
    command: "npm run lint"
    agents:
      queue: "lint" # 2-CPU, 4GB - cheap and fast enough

  - label: ":white_check_mark: Unit Tests"
    command: "npm test"
    agents:
      queue: "test-medium" # 8-CPU, 16GB

  - label: ":electric_plug: Integration Tests"
    command: "npm run test:integration"
    agents:
      queue: "test-large" # 16-CPU, 32GB, fast network

When running on cloud infrastructure with auto-scaling agent groups (using tools like the Buildkite Agent Stack for Kubernetes, or the AWS/GCP auto-scaler), this specialization also reduces cost: lint steps don't consume the expensive instance types, and the expensive instance types spin down when no integration tests are queued. Right-sizing agents is simultaneously a performance optimization and a cost optimization.

Optimization 5: Optimize Docker Build Layers

For teams using Docker in their pipelines - whether for building images or running tests in containers - Docker layer caching is one of the highest-impact optimizations available. The fundamental principle is that Docker rebuilds only layers that have changed and all layers above them in the Dockerfile. Structuring your Dockerfile to place slow, rarely-changing layers first and fast, frequently-changing layers last means most builds skip the slow work entirely.

The canonical antipattern is copying all source files before installing dependencies. Any change to any source file - including a README change - invalidates the dependency installation layer, forcing a full npm install or pip install on every build. The fix is always the same: copy only the dependency manifest first, install, then copy source.

# BEFORE: Source copy invalidates dependency layer on every change
FROM node:20-alpine
WORKDIR /app
COPY . .                  # ← Any file change invalidates everything below
RUN npm ci
RUN npm run build

# AFTER: Dependencies cached unless manifest changes
FROM node:20-alpine
WORKDIR /app
COPY package.json package-lock.json ./   # Only copy manifests first
RUN npm ci                               # ← Cached unless package-lock changes
COPY . .                                 # Source copy is fast (no re-install)
RUN npm run build

Beyond Dockerfile structure, consider using Docker BuildKit's --cache-from flag with a registry-backed cache. This allows agents to pull layer cache from a shared registry rather than relying on local disk - essential for ephemeral agents that don't persist between builds.

docker buildx build \
  --cache-from type=registry,ref=your-registry/your-image:buildcache \
  --cache-to type=registry,ref=your-registry/your-image:buildcache,mode=max \
  --push \
  -t your-registry/your-image:$BUILDKITE_COMMIT \
  .

Using mode=max exports all intermediate layers to the cache, not just the final image layers. This maximizes cache hit rates at the cost of additional registry storage.

Optimization 6: Eliminate Redundant Work with Conditional Steps

Not every step needs to run on every push. A documentation change should not trigger the full integration test suite. A change to a frontend component shouldn't rebuild and test backend services. Buildkite's if conditions and dynamic pipeline generation enable surgical targeting of the work that matters for a given change.

The if field on pipeline steps supports expressions evaluated against build metadata, including build.branch, build.tag, and - with some setup - the list of changed files. For more sophisticated file-based triggering, dynamic pipelines (where a script generates the pipeline YAML at runtime based on git diff) give complete control.

steps:
  # Only run full integration suite on main and release branches
  - label: ":electric_plug: Integration Tests"
    command: "npm run test:integration"
    if: build.branch == "main" || build.branch =~ /^release\//

  # Only run frontend tests if frontend files changed
  - label: ":art: Frontend Tests"
    command: "npm run test:frontend"
    if: build.message =~ /\[frontend\]/ || build.branch == "main"

For more precise change detection, use a dynamic pipeline approach: a first step runs a script that inspects git diff --name-only against the base branch, determines which services or packages were affected, and uploads a pipeline YAML that contains only the relevant steps. This pattern is particularly effective in monorepos where most changes touch only a small fraction of the codebase.

The productivity gain from conditional steps is asymmetric: you pay the overhead of the detection logic on every run (typically seconds), but you save the full step cost (potentially minutes) on all the runs where that step isn't needed.

Optimization 7: Tune Agent Concurrency and Resource Limits

Each Buildkite agent runs one job at a time by default. On agents with multiple CPUs, this means most of the machine sits idle while a single job occupies the agent. Enabling concurrent jobs (--acquire-job, or the job-concurrency config) allows multiple jobs to run simultaneously on the same agent, improving utilization for I/O-bound or low-CPU steps like lint, type-checking, and artifact uploads.

The right concurrency setting depends on job characteristics. Lint jobs are CPU-light and I/O-light - a 4-CPU agent can safely run 4 concurrent lint jobs. Test jobs that spawn browser instances or Docker containers may require exclusive CPU and memory access and should run at concurrency 1. The agent configuration supports per-queue concurrency limits, enabling this differentiation.

# Agent config for a lint-optimized agent
# /etc/buildkite-agent/buildkite-agent.cfg
name=lint-agent-%n
acquire-job=true
job-concurrency=4

# Resource limits prevent one job from starving others
# Use cgroup limits (systemd slice) or container resource constraints

There is a second dimension to resource tuning: memory and CPU limits on the processes spawned by jobs. Without limits, a single runaway test process (say, a memory leak in a test fixture) can exhaust agent resources and cause other concurrent jobs to fail with spurious OOM errors. Setting soft limits via ulimit or cgroup constraints in your agent startup script provides a safety boundary.

Optimization 8: Artifact Passing and Build Artifact Caching

In multi-step pipelines, later steps often need artifacts produced by earlier steps: compiled binaries, bundled assets, generated code. The naive approach is to rebuild from scratch at each step, which means your build step runs twice - once explicitly and once implicitly inside the deploy step. Buildkite's artifact system makes it straightforward to produce artifacts once and reuse them across steps.

Beyond pipeline-internal artifact passing, consider caching build outputs across pipeline runs. If your compiled output is deterministic - same source produces same binary - then a cache keyed on the source tree hash can skip the build entirely on re-runs, hot-fix branches, or cherry-picks that touch files outside the build inputs.

steps:
  - label: ":hammer: Build"
    key: "build"
    command: |
      npm run build
    artifact_paths: "dist/**/*"

  - label: ":rocket: Deploy Staging"
    depends_on: "build"
    command: |
      buildkite-agent artifact download "dist/**/*" .
      ./scripts/deploy.sh staging

A practical enhancement is to hash the source files contributing to a build artifact, check whether a matching artifact exists in your artifact store, and skip the build step entirely on a cache hit. This requires a small wrapper script around your build command but can eliminate build time entirely for commits that only change documentation, tests, or configuration files that don't affect the compiled output.

Optimization 9: Profile and Eliminate Slow Setup Scripts

Almost every pipeline accumulates setup cruft over time: environment variable configuration scripts, SDK installation steps, secrets injection logic, and infrastructure bootstrapping code that once took 5 seconds and now takes 90. This slow growth is invisible because it happens gradually and nobody owns the "total setup time" metric.

Profiling your pipeline setup time is straightforward. Add explicit timing to your agent hooks (pre-command, environment) and log the results as annotations or build metadata. Buildkite's job timeline view shows agent acquisition time, hook execution time, and command execution time separately - a useful starting point for identifying where time is going.

# In your pre-command hook: time each major setup block
time_block() {
  local name="$1"
  local start=$(date +%s%N)
  shift
  "$@"
  local elapsed=$(( ($(date +%s%N) - start) / 1000000 ))
  echo "~~~ :clock1: $name took ${elapsed}ms"
  buildkite-agent meta-data set "setup-time-${name}" "$elapsed"
}

time_block "secrets-injection" ./scripts/inject-secrets.sh
time_block "env-setup" ./scripts/setup-env.sh
time_block "sdk-check" ./scripts/ensure-sdks.sh

Common culprits found through profiling: tools installed at job start that should be baked into agent AMIs or Docker images; secrets management calls that make serial network requests (batching or pre-caching solves this); lockfile validation scripts that re-download index files on every run (disable or cache these); and AWS credential refresh logic that runs even on steps that don't use AWS.

The fix for most of these is the same: move slow, static setup work into agent images (AMI bake or Dockerfile), and reduce dynamic setup to only what genuinely changes per-job. The goal is a pre-command hook that completes in under 2 seconds.

Optimization 10: Use Pipeline-Level Concurrency Limits and Priority Queuing

As pipeline throughput grows, a new class of problem emerges: resource contention on shared infrastructure. If 50 engineers push simultaneously, 50 integration test suites might try to run at once against a shared staging database, causing flaky failures not from code bugs but from infrastructure overload. Buildkite's concurrency and concurrency_group settings address this directly.

The concurrency field limits how many instances of a particular step can run in parallel across all builds. The concurrency_group groups steps that share a resource under a single concurrency budget. This is the Buildkite-native solution to the "thundering herd on shared infrastructure" problem.

steps:
  - label: ":electric_plug: Integration Tests (against staging DB)"
    command: "npm run test:integration"
    concurrency: 3
    concurrency_group: "staging-db"
    # Only 3 integration test jobs will run simultaneously across ALL builds
    # Others queue and wait - avoiding staging DB overload

Priority queuing is the complement to concurrency limits. Main branch builds should take precedence over feature branch builds; hotfix pipelines should cut to the front of any queue. Buildkite supports priority on steps and on agents, enabling a coarse-grained priority system. Combined with agent auto-scaling that responds to queue depth, priority queuing ensures that the builds that matter most run fastest even under load.

Trade-offs and Pitfalls

Aggressive parallelism comes with coordination overhead. More parallel steps mean more agent starts, more artifact downloads, and more log streams to manage. For very short steps (under 30 seconds), the overhead of starting an agent, downloading the repository, and running the step can exceed the step's own execution time. There is a practical minimum granularity below which further parallelism hurts. A useful heuristic: if a step takes under 1 minute, think carefully before splitting it further.

Caching introduces a different class of risk: stale cache serving incorrect results. A cache keyed on a lockfile hash is safe for dependency caches, but a build artifact cache keyed on source files requires that your build is genuinely deterministic. Non-deterministic builds - those that embed timestamps, random UUIDs, or non-pinned dependency versions - will produce cache misses silently at the wrong times or, worse, cache hits that serve incorrect artifacts. Audit your build for non-determinism before implementing build artifact caching.

Conditional steps can also introduce subtle bugs. If your file-change detection incorrectly classifies a change as not affecting a service, tests that should have caught a regression never run. The safe default is to be conservative: err toward running more tests rather than fewer, and reserve aggressive skipping for low-risk, high-confidence cases (documentation changes, unrelated package updates). Always run the full suite on main branch regardless of what the diff looks like.

Best Practices for Sustained Pipeline Health

Fast pipelines are not a one-time achievement; they require ongoing stewardship. The most effective practice is tracking pipeline metrics as a first-class engineering concern. Export build duration, step duration, queue wait time, and cache hit rates to your observability platform (Datadog, Grafana, or Buildkite's own analytics). Set alerts on degradation - a pipeline that silently slows from 6 minutes to 12 minutes over three months is a common and preventable failure mode.

Treat your pipeline configuration as code, reviewed with the same rigor as application code. Pipeline changes that introduce new sequential dependencies, remove caching, or add expensive setup steps should be flagged in review just as any performance regression would be. Ownership matters: assign an engineer or a team to pipeline performance, give them a dashboard, and make their wins visible.

Review your agent fleet regularly against actual utilization data. Agent costs are often the largest CI/CD line item in cloud infrastructure budgets. An oversized agent fleet has the upside of zero queue wait time but pays for capacity that is idle most of the day. An undersized fleet saves money but creates queue pressure that negates all other optimizations. The right target is a fleet that scales dynamically to demand, using Buildkite's cloud-specific autoscalers (the Kubernetes agent stack, the AWS autoscaling integration, or the Buildkite Elastic CI Stack for AWS).

Finally, revisit your optimization decisions periodically. A caching strategy designed for a 50-test suite may be suboptimal for a 5,000-test suite. Parallelism settings that made sense on 4-core agents may leave resources on the table on 32-core agents. The optimizations above are a foundation, not a ceiling.

Key Takeaways

For teams looking to apply these techniques immediately, here are five steps with the highest expected return on investment:

1. Audit your dependency graph and remove unnecessary depends_on relationships. Most pipelines contain at least one step that was serialized out of habit, not necessity. Removing even one sequential constraint can cut wall time by the duration of that step.

2. Enable parallelism on your test suite with a split count of 4 or 8. This single change often delivers the largest single percentage improvement. Start with an even split and refine to time-based splitting once you have timing data.

3. Implement dependency caching keyed on your lockfile hash. A well-implemented dependency cache eliminates the most consistently slow part of most pipelines on every cache-hit run.

4. Restructure your Dockerfiles so dependency installation comes before source copy. This is a 30-minute change with durable impact, as it improves every Docker-based step on every push.

5. Add pipeline duration tracking to your observability stack. You cannot improve what you do not measure, and pipeline slowdowns that happen gradually are invisible without metrics.

80/20 Insight

Of the ten optimizations above, three produce the majority of the improvement in most pipelines: parallelism (both step-level and within-step), dependency caching, and removing unnecessary sequential constraints. These three target the core mathematical limits of pipeline speed - the critical path length and the fraction of time spent on repeated work. The remaining optimizations are refinements that compound the gains from these three.

If you have limited time, instrument your pipeline to find the longest sequential chain and the most frequently repeated expensive work. Parallelize the first, cache the second. That approach, applied thoughtfully, gets most teams to 50-60% improvement before they've touched agent sizing, Docker optimization, or concurrency controls.

Conclusion

CI pipeline performance is an underinvested area in most engineering organizations. The work is unglamorous, the wins are diffuse (every engineer benefits a little rather than one team benefiting a lot), and the impact is hard to quantify without metrics. But the aggregate impact of fast pipelines on developer velocity, deployment frequency, and team morale is substantial and well-documented.

Buildkite's architecture - with its separation of orchestration and execution, its flexible agent model, and its rich pipeline DSL - provides more optimization surface than most CI platforms. The techniques above are not theoretical: they are patterns that appear repeatedly in high-throughput Buildkite deployments, and they compose well. Implement them incrementally, measure the impact of each, and use the data to guide the next investment.

The goal is not a 6-minute pipeline for its own sake. The goal is a feedback loop fast enough that developers stay in flow, confident enough that teams push to production without anxiety, and stable enough that speed doesn't come at the cost of reliability. With deliberate optimization, that combination is achievable.

References

  1. Forsgren, N., Humble, J., & Kim, G. (2018). Accelerate: The Science of Lean Software and DevOps. IT Revolution Press.
  2. Buildkite Documentation - Pipeline Configuration Reference. https://buildkite.com/docs/pipelines
  3. Buildkite Documentation - Agent Configuration. https://buildkite.com/docs/agent/v3/configuration
  4. Buildkite Documentation - Artifacts. https://buildkite.com/docs/agent/v3/artifacts
  5. Buildkite Documentation - Concurrency and Parallelism. https://buildkite.com/docs/pipelines/controlling-concurrency
  6. Buildkite Documentation - Test Analytics (Test Collector). https://buildkite.com/docs/test-analytics
  7. Buildkite Elastic CI Stack for AWS. https://github.com/buildkite/elastic-ci-stack-for-aws
  8. Docker Documentation - BuildKit Cache. https://docs.docker.com/build/cache/
  9. Docker Documentation - Dockerfile Best Practices. https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
  10. pytest-split - Distributed test suite splitting plugin. https://github.com/jerry-git/pytest-split
  11. Jest - --shard CLI option. https://jestjs.io/docs/cli#--shard
  12. Kim, G., Behr, K., & Spafford, G. (2013). The Phoenix Project. IT Revolution Press.