Finding and Fixing Flaky GitHub Actions Jobs

A flaky GitHub Actions job passes and fails on the same commit with no code change in between, and the fix depends on whether the cause sits in the test suite or in the machine under it. Rank jobs by success rate per repository, workflow, and job name in the WarpBuild Jobs report, read the CPU and memory percentiles on the same row, and those two cases separate before anyone opens a stack trace.

This guide covers the four flake sources worth separating, the report columns that tell resource starvation apart from genuine nondeterminism, the workflow change that opens a shell on the runner that produced the failure, and a weekly cost model for living with a flake against hunting it down.

Diagnosis

Start from a ranked list rather than from the last complaint in Slack. A repository with 400 distinct workflow jobs usually has under ten that are genuinely unstable, and the loudest ones are often stable jobs failing for a real reason.

The four sources worth separating

Shared state between tests. A fixture, a database row, a temp file, a global mock, or an environment variable survives one test and changes the next. The signature is that failures move around: the same commit fails on checkout.spec one run and billing.spec the next. Parallel workers make this worse because two tests now touch the same row at once instead of in sequence.

Timing dependence. The test asserts on something that becomes true a few hundred milliseconds after it looks. Any suite carrying sleep 2 or a fixed wait already contains this bug; the sleep hides it on a fast machine and exposes it on a loaded one. This source is the one most often blamed on the runner, because a busy machine widens every window.

Resource exhaustion on an undersized runner. The job is deterministic and the machine runs out of memory, disk, or CPU headroom part way through, so the result depends on how much other work landed on the same box. A Jest suite spawning workers from os.cpus() behaves differently on a 2 vCPU, 8 GB runner than on a 16 vCPU, 64 GB one. Retry logic in the test framework leaves this case exactly where it was, because the suite is deterministic and the machine is the part that varies.

Network calls to third parties. Package registries, container registries, SaaS APIs, and OAuth endpoints all return 429 and 502 under their own load, on their own schedule. The tell is clustering: several unrelated jobs go red inside the same ten minutes, then everything recovers.

Rank by success rate before touching anything

The WarpBuild Reports page aggregates every job into one row per unique combination of repository, workflow, and job name. Each row carries run count, success rate, duration P75 and P90, queue time P75 and P90, CPU P75 and P90, and memory P75 and P90 over the selected date range.

Two moves turn that table into a flake list:

• Sort by success rate ascending and ignore every row with a run count under 20. A job that ran three times and failed once tells you nothing yet.
• Filter by repository, workflow, job name, runner label, or stack to keep one team's surface in view, then export the filtered rows to CSV. The export carries every row matching the current filters and sort order rather than the visible page.

Success rate alone still mixes real bugs into the list, so a job that started failing on every run after a merge shows up next to a job that fails one run in fifteen. The time-series chart separates them: pick a metric and a percentile, and each job on the current table page draws its own line, so a step change on a single date reads differently from a flat noisy band.

Correlate with CPU and memory percentiles

The CPU and memory columns are what make this report a flake tool rather than a dashboard. WarpBuild flags an instance as under-provisioned when max sustained CPU, memory, or filesystem utilization reaches 80 percent, and the P90 columns on the Jobs row surface that same pressure aggregated across runs.

A dash in those columns means the job ran without telemetry, so CPU and memory require observability to be enabled before this correlation works at all.

Fix

Confirm the classification before changing code, because a fix applied to the wrong source leaves the flake in place and adds a retry that hides it.

Confirm by repeating one commit

Run the suspect job against a single commit thirty times. Thirty repetitions catch a failure that appears once in twenty runs with margin, and a failure that survives thirty green runs is rare enough that the next step is instrumentation instead of a fix.

Two settings make the repeat useful. fail-fast: false keeps the remaining repetitions running after the first red one, which is the evidence you are collecting. And there are no hard concurrency caps on any WarpBuild tier, with capacity that adjusts dynamically, so thirty repetitions start together instead of draining through a fixed pool over an afternoon.

Fix by source

Shared state is fixed in the suite: give each test its own database schema or transaction, generate unique fixture keys per test, reset global mocks in an afterEach, and run the suite with a shuffled seed in GitHub Actions so ordering assumptions fail early and loudly rather than once a month.

Timing dependence is fixed by replacing every fixed sleep with a wait on the actual condition, and by giving those waits a timeout long enough to survive a loaded machine. A wait that polls for the condition costs nothing when the condition is already true.

Resource exhaustion is fixed by moving runs-on to a larger label, or by capping worker counts so the suite stops sizing itself from the machine. Both are one-line changes and both should be verified by re-reading the CPU and memory percentiles after a day of runs.

Third-party network calls are fixed by removing them from the test path. Mock the external service, pin and cache dependency downloads so the registry is touched once, and keep genuine integration checks in a separate scheduled workflow where a 429 does not block a pull request.

Get on the runner that produced the failure

When repetition reproduces the failure but the logs do not explain it, stop guessing and take a shell. The Action Debugger is a free, open-source GitHub Action that pauses a workflow and opens an SSH session on the runner machine. Add it to the job guarded by failure():

      - name: Setup interactive ssh session
        if: ${{ failure() }}
        uses: Warpbuilds/[email protected]
        with:
          limit-access-to-actor: true
        timeout-minutes: 25

Execution pauses as soon as that step is invoked. The SSH URL is written to the action logs and posted as a check on the corresponding GitHub run, and the workflow stays paused until someone connects and exits the session.

Two details matter more here than on an ordinary failure. First, security: by default, if the GitHub user who triggered the run has SSH keys on their account, only that user can connect, and if they have no keys on the account, anyone who has or guesses the generated SSH URL can connect to the session. Setting limit-access-to-actor: true forces the restriction. Deterministic URLs through named sessions require an API key from WarpBuild support and also require limit-access-to-actor: true.

Second, posting the URL as a check needs write permission on the checks scope. Under repository Settings, Actions, General, set Workflow permissions to read and write permissions. On many organizations that setting has to change at the organization level, because the organization value overrides the repository one.

Once connected, the questions worth asking are the ones a re-run cannot answer: free -m and df -h for headroom at the moment of failure, ps aux --sort=-%mem | head for the process that grew, the test database state left behind by the previous test, and the failing command re-run by hand with one flag changed at a time.

Configuration

Here is the full flake-hunting workflow. It repeats one commit thirty times, keeps every repetition alive, and drops into a shell on whichever repetition goes red.

name: flake-hunt
on:
  workflow_dispatch:
    inputs:
      test_filter:
        description: Test name pattern to repeat
        required: false
        default: ""

jobs:
  repeat:
    strategy:
      fail-fast: false
      matrix:
        run: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
              11, 12, 13, 14, 15, 16, 17, 18, 19, 20,
              21, 22, 23, 24, 25, 26, 27, 28, 29, 30]
    runs-on: warp-ubuntu-latest-x64-4x
    permissions:
      contents: read
      checks: write
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - name: Run suite with shuffled order
        run: npm test -- --shard=1/1 --randomize --testNamePattern "${{ inputs.test_filter }}"
      - name: Setup interactive ssh session
        if: ${{ failure() }}
        uses: Warpbuilds/[email protected]
        with:
          limit-access-to-actor: true
        timeout-minutes: 25

Sizing is the fix for the starvation case, so keep the Linux x64 catalog in front of you when you change runs-on:

Rates come from the WarpBuild pricing page and are billed per minute, checked on 2026-08-13.

Which matters when a flake only appears on one of them. Swap the single runs-on above for a platform axis and the thirty repetitions become a platform sweep across warp-ubuntu-latest-x64-4x, warp-ubuntu-latest-arm64-4x, warp-macos-15-arm64-6x, and warp-windows-latest-x64-4x, which narrows an intermittent failure to one platform in a single dispatch.

If your runners live in your own cloud account, BYOC runs on AWS, GCP, and Azure, and Terraform support exists for BYOC on AWS, so the instance shape behind a starving job is described in code you can diff against the shape you intended. Access to the reports follows your WarpBuild organization membership, and SSO is available for a flat $250 per month, whatever the user count.

Cost or Time Model

Flakes are usually argued about in dollars and decided by engineer hours. Here is both, on one job.

Assumptions, stated so you can substitute your own:

• The job runs 200 times per week on warp-ubuntu-latest-x64-4x at $0.008 per minute, from the WarpBuild pricing page, checked on 2026-08-13.
• Each run bills 12 minutes.
• The Jobs report shows a 92 percent success rate, so 16 runs per week go red without a code cause.
• Every spurious failure is re-run once, and one of those re-runs fails again.
• A human loses 12 minutes of waiting plus 5 minutes of context switching per spurious failure.

The runner-cost column moves by $1.63 per week, which pays back the $3.08 investigation in two weeks and then stops being interesting. The column that decides it is the last one: 272 engineer minutes per week is about 4.5 hours, and that is one job. Ten such jobs across a repository is a full engineering week spent watching re-runs.

The starvation case has its own arithmetic, and it does not always come out cheaper. Moving a job from warp-ubuntu-latest-x64-2x at $0.004 per minute to warp-ubuntu-latest-x64-4x at $0.008 per minute doubles the per-minute rate, so 2,400 billed minutes move from $9.60 to $19.20 per week. What you buy is the 204 minutes of re-run time and the 272 engineer minutes above, which is the trade most teams take.

Two costs are easy to forget on the debugging side. Runner minutes bill for as long as a debug session stays open, so a 25 minute session on warp-ubuntu-latest-x64-4x costs $0.20 and an abandoned one runs until GitHub kills the workflow after 6 hours, at $2.88. Put timeout-minutes on every debugger step. And the thirty-repetition matrix bills all thirty jobs, which is the $2.88 line in the hunt row above, so run it against one narrowed test pattern rather than the whole suite.

For the failure classes that are not flakes at all, start at how to debug a failed GitHub Actions job. If the complaint is duration rather than instability, the same report columns answer it: see GitHub Actions build duration percentiles. For the shell access itself, see can I SSH into a GitHub Actions runner. Browser suites carry their own flake profile, covered in running Playwright tests on GitHub Actions.

FAQ

How do I tell a flaky test from an undersized runner?

Open the Jobs report, find the row for the repository, workflow, and job name, and read success rate beside CPU P90 and memory P90. A job with a success rate under 98 percent and a memory P90 at or above 80 percent is starving, and a larger runner label fixes it. The same success rate with CPU and memory P90 both low points at shared state, timing, or a third-party network call.

How many times should I re-run a commit to confirm a flake?

Thirty repetitions of the same commit catches a failure that shows up once in twenty runs with room to spare, and a fail-fast: false matrix keeps every repetition running after the first red one. There are no hard concurrency caps on any WarpBuild tier, so the thirty jobs start together instead of draining through a fixed pool.

Can I SSH into the runner that produced the flaky failure?

Yes. Add a Warpbuilds/[email protected] step guarded by if: ${{ failure() }} and the workflow pauses on the failing run, prints an SSH URL in the job logs, and posts it as a check. Set limit-access-to-actor: true, because without it and without SSH keys on the triggering account, anyone holding the generated URL can connect.

Does an open debug session keep billing runner minutes?

Yes. Runner minutes are billed for as long as the session stays open and GitHub kills workflows after 6 hours by default, so put timeout-minutes on the debugger step. At the warp-ubuntu-latest-x64-4x rate of $0.008 per minute from the WarpBuild pricing page, checked on 2026-08-13, a 25 minute session costs $0.20 and a forgotten 6 hour one costs $2.88.