How Do I Find Which Workflow Got Slower?

Find the workflow that got slower by pulling duration percentiles per job for two date ranges and ranking the jobs by the change in the tail percentile, because a tail move is the signal an average hides. The WarpBuild Jobs report carries Duration P75 and P90 for every unique repository, workflow, and job name combination, and it supports date range selection and CSV export, so the whole comparison is two exports, a join, and a subtraction.

Answer

Rank on the tail, then rank on volume. A percentile that moved five minutes on a job that runs 60 times a month costs less total waiting than a percentile that moved one minute on a job that runs 1,200 times, and both readings come out of the same two exports.

Here is why the tail is the field to diff. Take a unit test job with 1,240 runs in a month. Suppose the slowest tenth of those runs, 124 of them, each gained 5.7 minutes because a new integration test started retrying. The mean absorbs 124 x 5.7 minutes spread across 1,240 runs, which is 0.57 minutes, or 34 seconds. Duration P90 reports the same regression as 5.7 minutes, because P90 sits inside the group that moved.

These are the fields the Jobs section reports per unique repository, workflow, and job name, per the reports documentation.

A worked comparison across two months, with example figures to replace with your own two exports. Both period columns come from the same table with the date range moved, durations are in minutes, and the last column multiplies the P90 change by the later month's run count to rank the regressions by total added wait.

Three readings fall out of that table. The nightly job has the largest per-run regression at 11.5 minutes and ranks third on total wait, because it runs 62 times. The ci.yml unit test job held P75 almost flat while P90 gained 5.7 minutes, which describes a subset of runs going long rather than every run getting heavier. The Docker build job moved on both percentiles, which describes work added to every run.

Detail

The export procedure

1. Open the Jobs section of Reports and set the date range to the earlier month. Filter by repository if the account holds more than one.
2. Export the table to CSV. The export includes every row matching the current filters and sort order rather than the visible page.
3. Set the date range to the later month and export again.
4. Join the two files on repository, workflow, and job name. Rows that appear in one file only are new or retired jobs and belong in a separate list.
5. Compute the P90 delta per row, multiply by the later run count, and sort by that product.
6. Read Queue Time P75 and P90 on the top rows before you read anything else.

Step 6 decides which half of the problem you have. Queue wait sits before the job starts and is reported separately in the Queue Timings section per runner label and stack, so a job whose queue time grew while its duration held flat is a capacity story with no slow step to find.

WarpBuild provides Linux x64, Linux ARM64, macOS, and Windows runners, and the Jobs table aggregates all four in one place, so a regression confined to one platform ranks against everything else rather than hiding in a separate tool.

Date the bend, then open the run

The time-series chart plots one metric, duration or queue time or CPU or memory, at P75 or P90 over time, with one line per job on the current table page. That is the surface that turns "July was worse" into a week, and often into a specific merge.

Once you have the week, the next move is step level. Pull the jobs of one run from before the bend and one run from after it through the GitHub REST API and subtract started_at from completed_at for each step, using the workflow jobs endpoint. The ordered procedure is in the guide to finding the slow step, and the percentile background is in the guide to build duration percentiles.

If the step timings look the same on both runs, the machine is the remaining variable. CPU and memory percentiles in the Jobs table need observability enabled, and telemetry is collected only for jobs longer than about one minute, so short jobs show a dash in those columns. The Usage view puts utilization charts and GitHub Actions logs on one timeline for a named job. CI observability is part of the WarpBuild product surface alongside snapshot runners, remote Docker builders, an MCP server, and the Action Debugger.

Tracking these deltas on a schedule instead of on demand is what GitHub Actions observability covers, and turning a percentile into a target your team commits to is covered in the guide to CI SLOs and build time budgets.

What the regression costs

Put a number on the top row before deciding whether to chase it. The unit test job above added 124 x 5.7 = 706.8 minutes of execution across July. At warp-ubuntu-latest-x64-4x, which is $0.008 per minute on the pricing page, that is $5.65 on the invoice. The same 706.8 minutes is about 11.8 hours of engineers waiting on red or yellow checks.

The bill and the wait move at different scales, which is the point of running both columns. For reference on the rate, the 4-core Linux larger runner is listed at $0.012 per minute in GitHub's published Actions prices, checked on 2026-08-13.

Related Questions

Why does average build time stay flat when a workflow gets slower?

Because the mean spreads a regression across every run. If the slowest tenth of 1,240 runs each gained 5.7 minutes, the mean moves by 0.57 minutes while duration P90 moves by the full 5.7. The tail percentile reports the regression at the size the person waiting on it experiences, which is the argument for tracking it in the guide to build duration percentiles.

Can I compare percentiles other than P75 and P90?

The Jobs report carries Duration P75 and P90 and Queue Time P75 and P90 per unique repository, workflow, and job name. For any other percentile, pull per-run job records from the GitHub REST API, subtract started_at from completed_at, and compute the percentile you want from the raw list.

Duration moved but the workflow file did not change. Where do I look next?

Read queue time P75 and P90 on the same row first, because queue wait sits outside the steps and no workflow edit changes it. If queue time held flat, open the time-series chart to date the bend, then rank the steps of a run from that week against a run from before it using the guide to finding the slow step.