Build Duration Percentile

A build duration percentile is the duration below which a stated share of builds finish, written as p50, p95, or p99. A p95 build duration of twenty minutes means 95 percent of the runs in the window completed in twenty minutes or less and the remaining 5 percent took longer.

Percentiles are reported because the average of a build duration series is pulled upward by a small number of slow runs while still landing below all of them. The slow tail is the part a developer sits through, and it is the part an average erases.

Definition

Take every run of one job over a stated window, sort the durations from shortest to longest, and read off the value at a rank. The nearest-rank method puts the p-th percentile at index ceil(p / 100 * N) in that sorted list, where N is the number of runs. p50 is the median, p95 is the value with 5 percent of runs above it, and p99 is the value with 1 percent above it.

Four choices decide what the resulting number actually describes.

The unit being measured. A duration can be a step, a job, or a whole workflow run. A workflow run holds jobs that execute in parallel, so its duration is the critical path through the dependency graph rather than the sum of its jobs. Percentiles taken at different units are not interchangeable, and the p95 of a workflow is unrelated to the sum of the p95s of its jobs.

Where the clock starts. A number measured from the moment the job was queued includes time spent waiting for a machine. A number measured from the first step includes only execution. Those two series diverge under load, and mixing them turns a capacity problem into what looks like a slow test suite. See queue time for the interval that sits between them.

The estimation method. Nearest-rank returns a value that actually occurred. Linear interpolation returns a value between the two neighboring runs. On the same 20 runs used below, nearest-rank gives a p95 of 19:48 and linear interpolation gives 20:01. The difference is small at p95 with a healthy sample and grows as the sample shrinks.

The population. Percentiles do not average and do not add. Two jobs each with a p95 of 10 minutes do not combine into a fleet p95 of 10 minutes, and a weekly p95 cannot be recovered from seven daily p95 values. The underlying durations have to be re-ranked over the new window.

Sample size sets a floor on what a percentile can resolve. With 20 runs the p95 lands on the second slowest run, so a single unlucky build swings it. With 100 runs it lands on the fifth slowest. Tooling that reports p75 and p90 rather than p95 is making the same tradeoff in the other direction: a lower percentile is steadier on a thin sample and hides less of the tail than an average does. A p95 cannot be derived from a p75 and a p90, so the percentile a report exposes is the one a budget has to be written against.

Example

This workflow writes one line per run into the job summary: the elapsed seconds and whether the dependency cache was restored. Those lines are what a percentile is computed from later.

name: test
on:
  push:

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - name: Mark start
        id: start
        run: echo "epoch=$(date +%s)" >> "$GITHUB_OUTPUT"
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - name: Restore npm cache
        id: npm-cache
        uses: actions/cache@v4
        with:
          path: ~/.npm
          key: npm-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
      - run: npm ci
      - run: npm test
      - name: Record duration
        if: always()
        env:
          START: ${{ steps.start.outputs.epoch }}
          CACHE_HIT: ${{ steps.npm-cache.outputs.cache-hit }}
        run: |
          seconds=$(( $(date +%s) - START ))
          echo "job=unit-tests seconds=$seconds cache_hit=${CACHE_HIT:-false} run=${{ github.run_id }}" \
            >> "$GITHUB_STEP_SUMMARY"

The clock starts at the first step, so this series measures execution and leaves queue time out. Recording cache_hit on the same line is what makes the split further down possible.

Twenty consecutive runs of that job, sorted shortest to longest:

Nearest-rank puts p50 at index ceil(0.5 * 20), which is rank 10, giving 4:02. It puts p95 at index ceil(0.95 * 20), which is rank 19, giving 19:48. The arithmetic mean of the same twenty runs is 6:40, which is higher than fifteen of the twenty runs and still under a third of the p95. Quoting 6:40 as the build time misrepresents both ends of the series.

The gap between 4:02 and 19:48 is a ratio of about 5 to 1, and a spread that wide says the same steps are running under conditions that change from run to run. Identical work on a stable machine with stable inputs produces a tight distribution, so a fanned-out tail points at something outside the test code: a cache that sometimes misses, a shared machine under contention, a network dependency that occasionally retries, or two different populations of runs recorded under one label.

The cache_hit column identifies which one it is here. Splitting the same twenty runs by that field:

Each population is tight on its own. The combined p95 of 19:48 was describing a mixture, and the fix it points at is cache key design rather than test performance. The five-run population also shows the sample size limit directly: its p95 lands on the slowest run it has, so that 24:10 is an upper bound on a thin sample rather than a stable figure.

Two habits follow. Split a duration series by anything that changes the work before reading a percentile from it, including matrix variant, runner label, event type, and cache outcome. And track p50 and p95 side by side, because the p50 tracks the common path while the p95 tracks the run a developer is waiting on when they ask why the build is slow.

Related Terms

• Reading build duration percentiles out of GitHub Actions run history: pulling per-job durations from run history and turning them into p50 and p95 series per workflow.
• Setting p50 and p95 build time budgets per workflow class: choosing targets per workflow class, and the escalation rule for a broken budget.
• Queue time and why it sits outside the duration measure: the interval between a job being queued and a runner starting it, which is a separate series with its own percentiles.
• GitHub Actions job summaries: the upstream reference for the $GITHUB_STEP_SUMMARY file used in the example.
• WarpBuild reports documentation: per-job duration and queue-time percentiles with CSV export.
• WarpBuild observability documentation: CPU, memory, and disk utilization per runner instance, which is where a wide duration spread gets checked against resource limits.
• WarpBuild pricing: per minute rates by runner type.