Flaky Test

A flaky test is a test that passes on one execution and fails on the next against the same code, with nothing in the source changing between the two. The result is nondeterministic because the test depends on something the code under test does not control: elapsed time, state that another test left behind, or the order the suite happened to run in.

Flakiness is a property of the test itself. A busier or slower machine changes how often a latent flake surfaces, so one suite can look stable on a laptop and unstable in GitHub Actions while the defect sits in the test the whole time.

Definition

A test is flaky when repeated executions against an identical commit, an identical dependency lockfile, and an identical environment produce both pass and fail outcomes. That wording sets the boundary on either side. A test that fails on every execution is a failing test and points at a real defect. A test that passes on every execution is a passing test. Flakiness occupies the region between those two, and it is measured as a rate rather than reported as a state.

The flake rate of a test is the fraction of executions on one commit that fail. A test that fails 3 executions out of 100 has a flake rate of 3 percent. Because the rate is a probability, one green re-run says almost nothing about the test, and one red run establishes only that the rate sits above zero.

The three usual mechanisms

Timing. The test asserts on a condition that becomes true shortly after it looks. Any suite carrying a fixed sleep already holds this defect, because the sleep hides it on an idle machine and exposes it on a loaded one. Real clocks add their own variants, including tests that break at a month boundary or across a daylight saving change.

Shared state. Two tests read and write the same thing. The write from one lands between the write and the assertion of the other. Parallel execution turns this from a rare event into a frequent one, because work that used to happen in sequence now overlaps.

Order dependence. A test relies on setup performed by a test that runs before it. The suite stays green while the ordering holds and turns red as soon as a framework shuffles the sequence, shards the suite across workers, or somebody deletes the earlier test.

Calls to services outside the repository sit slightly apart from those three. A rate limit, a DNS failure, or a 502 from a third-party host produces the same pass-and-fail pattern, and the repair is a fake or a recorded response rather than a change to the test's own logic.

Why a single re-run is weak evidence

Re-running is the cheapest response to a red build and the weakest evidence about it. Repetition is what turns a rate into a number. The table below assumes independent executions and gives the chance a suite survives a fixed number of repetitions.

Two readings fall out of that arithmetic. A green re-run is the expected outcome at every rate in the table, so it carries very little information on its own. And 30 repetitions surface a 5 percent flake about four times out of five, which is why a repetition matrix is the standard way to confirm one.

Example

The clearest flake to reproduce is a shared fixture under parallel execution. This workflow runs a Python suite across four workers:

name: test
on: [push]

jobs:
  unit-tests:
    runs-on: warp-ubuntu-latest-x64-4x
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest -n 4 tests/

Two tests in that suite share one session-scoped fixture, and each resets the balance on it before acting:

# tests/conftest.py
@pytest.fixture(scope="session")
def account():
    return Account.objects.get_or_create(id=1)[0]

# tests/test_billing.py
def test_charge(account):
    account.set_balance(100)
    charge(account, 40)
    assert account.balance == 60

# tests/test_refunds.py
def test_refund(account):
    account.set_balance(100)
    refund(account, 25)
    assert account.balance == 125

With a single worker both tests pass every time, because each one resets the row, acts on it, and asserts before the next test starts. Add -n 4 and the two tests execute at the same moment against the same row. When test_charge calls set_balance(100) after test_refund has already applied its refund, the refund assertion reads 100 instead of 125:

FAILED tests/test_refunds.py::test_refund - assert 100 == 125

Which of the two reports the failure depends on which one loses the race, so the failing test name moves between runs. That is the shared-state signature from the table above, and it is the detail that separates this from a genuine defect in the refund code.

Two checks confirm the diagnosis. Running the file on its own with pytest tests/test_refunds.py passes on every attempt, which clears the refund logic. Running the whole suite with pytest -n 0 tests/ passes as well, which places the cause in the parallel execution rather than in either assertion.

Confirming the rate with a repetition matrix

A matrix runs the same commit many times and reports every result. That needs fail-fast set to false, so one red combination leaves the remaining combinations running instead of cancelling them (GitHub workflow syntax reference, checked on 2026-08-13):

jobs:
  repeat:
    strategy:
      fail-fast: false
      matrix:
        attempt: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
    runs-on: warp-ubuntu-latest-x64-4x
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: pytest -n 4 tests/test_billing.py tests/test_refunds.py

Ten attempts against a suite whose per-execution flake rate is 20 percent come back all green about 11 percent of the time, since 0.8 raised to the tenth power is 0.107. Ten attempts are therefore plenty for an obvious flake and thin for a rare one. Widen the matrix until attempts multiplied by the estimated rate predicts at least a few failures.

The repair for this particular case is to stop sharing the row. Give each test its own record, scope the fixture to the function instead of the session, or key the record on the worker ID that the parallel plugin exposes to each process.

Related Terms

• Find and fix flaky GitHub Actions jobs: how to rank jobs by success rate, tell resource starvation apart from genuine nondeterminism, and open a shell on the machine that produced the failure.
• Fail fast in a GitHub Actions matrix: the matrix setting that cancels the remaining combinations after the first failure, and the reason a repetition matrix turns it off.
• How to retry a failed step in GitHub Actions: the retry options available at the step level and what a retry hides about the underlying rate.
• GitHub workflow syntax reference for strategy.fail-fast: the documented behavior of the matrix setting used in the repetition example.
• WarpBuild reports documentation: success rate, duration percentiles, and queue time percentiles per repository, workflow, and job name.
• Action-Debugger documentation: the open source action that pauses a workflow on a step and opens an SSH session on the runner.
• WarpBuild pricing: per minute rates by runner type.