How to detect flaky tests

A single green run proves nothing. Loop the file with npx vitest run twenty times and count the results. On the project below one run of twenty fails a deadline assertion, and three such loops gave four, two and one failures. A rate, not a verdict, identifies a flaky test.

Why check this

Run this when a pipeline goes red and green again on a rerun with no commit in between, and before adding --retry to a CI job, because that flag turns the evidence off.

The failure it prevents is a team's trust in its own suite. A test that fails one run in ten teaches everybody to press rerun, and after a month nobody reads a red build closely enough to notice a real regression next to the usual offender. Naming the unstable test and its rate is what lets you quarantine it instead of ignoring the pipeline.

Flakiness is a rate, and a rate needs a sample. The steps below take one test from "it passed" to "it fails two runs in twenty, always on line 16, always within 12 ms of its deadline".

Prerequisites

• Node 22 and npm. Every figure came from Node 22.23.2 with Vitest 5.0.0, on one Windows 11 machine, on 2026-09-12.
• A POSIX shell for the loops. The commands below ran in Git Bash.
• A project with a known unstable test. Build this one in an empty directory outside your repository. The test finishes the export before the deadline races real work against a 25 ms timer, the commonest shape of a flaky unit test. See the Vitest CLI reference.

mkdir covdemo && cd covdemo
npm init -y > /dev/null
npm pkg set type=module
npm install -D vitest@5
mkdir src test

cat > src/export-job.js <<'EOF'
export function makeRows(n) {
  const rows = [];
  for (let i = 0; i < n; i += 1) rows.push({ sku: 'A' + i, qty: (i % 7) + 1, price: (i % 13) + 0.5 });
  return rows;
}

export async function exportRows(rows) {
  const out = [];
  for (const row of rows) {
    out.push(`${row.sku},${row.qty},${(row.qty * row.price).toFixed(2)}`);
    if (out.length % 2000 === 0) await new Promise((r) => setImmediate(r));
  }
  return out.join('\n');
}
EOF
cat > test/export-job.test.js <<'EOF'
import { describe, it, expect } from 'vitest';
import { makeRows, exportRows } from '../src/export-job.js';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

describe('exportRows', () => {
  it('writes one line per row', async () => {
    const csv = await exportRows(makeRows(3));
    expect(csv.split('\n')).toHaveLength(3);
  });

  it('finishes the export before the deadline', async () => {
    const rows = makeRows(20000);
    const deadline = sleep(25).then(() => 'deadline');
    const result = await Promise.race([exportRows(rows), deadline]);
    expect(result).not.toBe('deadline');
  });

  it('waits for the upload to settle', async () => {
    await sleep(700);
    expect(await exportRows([])).toBe('');
  });
});
EOF

• The rate belongs to the machine as much as to the test. Three loops of 20 on this file gave 4 failures under load, then 2 and 1 when idle. Compare rates from one sitting.

Steps

1. 1

   Step 1.

   Run the file once and note that a green result answers nothing.

   npx vitest run test/export-job.test.js
   

    RUN  v5.0.0 C:/…/covdemo
   
   Test Files  1 passed (1)
       Tests  3 passed (3)
    Duration  937ms (tests 94%, transform 3%, import 2%, worker 1%)

2. 2

   Step 2.

   Run the same file twenty times and count the outcomes.

   for i in $(seq 1 20); do npx vitest run test/export-job.test.js >/dev/null 2>&1 && echo pass || echo fail; done | sort | uniq -c
   

         1 fail
      19 pass

   One in twenty is a rate, poorly measured. Two earlier loops on the same code gave four failures and two: 7 in 60 runs, and the reason the test reached the main branch.

3. 3

   Step 3.

   Run the file again until one run goes red, and read that run instead of rerunning past it.

   npx vitest run test/export-job.test.js
   

    ❯ test/export-job.test.js (3 tests | 1 failed) 757ms
    ❯ exportRows (3)
      × finishes the export before the deadline 37ms
   
   FAIL  test/export-job.test.js > exportRows > finishes the export before the deadline
   AssertionError: expected 'deadline' not to be 'deadline' // Object.is equality
   ❯ test/export-job.test.js:16:24
      14|     const deadline = sleep(25).then(() => 'deadline');
      15|     const result = await Promise.race([exportRows(rows), deadline]);
      16|     expect(result).not.toBe('deadline');
        |                        ^
      17|   });
      18|
   
   Test Files  1 failed (1)
       Tests  1 failed | 2 passed (3)
    Duration  1.12s (tests 89%, transform 5%, import 5%, worker 1%)

   The duration next to the test name is the evidence: 37 ms against a 25 ms deadline. Failing 12 ms over budget is a timer race, not a logic bug.

4. 4

   Step 4.

   Turn on retries and count how often they rescue the run.

   for i in $(seq 1 30); do if npx vitest run test/export-job.test.js --retry=2 --reporter=verbose > run.log 2>&1; then grep -q 'retry x' run.log && echo "green after a retry" || echo "green first try"; else echo "red after 3 attempts"; fi; done | sort | uniq -c
   

        29 green first try
       1 red after 3 attempts

   Three attempts in one process are not three independent samples. This test failed 7 of 60 plain runs, so independence predicts a triple failure about once in 600. Measured over 50 retried runs: 3.

5. 5

   Step 5.

   Run the verbose form until one green run reports a retry, and see how it is recorded.

   npx vitest run test/export-job.test.js --retry=2 --reporter=verbose
   

    RUN  v5.0.0 C:/…/covdemo
   
   ✓ test/export-job.test.js > exportRows > writes one line per row 4ms
   ✓ test/export-job.test.js > exportRows > finishes the export before the deadline 59ms (retry x1)
   ✓ test/export-job.test.js > exportRows > waits for the upload to settle 715ms
   
   Test Files  1 passed (1)
       Tests  3 passed (3)
    Duration  1.09s (tests 91%, transform 5%, import 3%, worker 1%)

   The summary says Tests 3 passed (3) for a run in which this test failed once. Only the (retry x1) marker records it, and the default reporter prints no per-test line for a passing file.

6. 6

   Step 6.

   Change the deadline from 25 ms to 30 ms in the test, then measure the new rate over fifty runs.

   for i in $(seq 1 50); do npx vitest run test/export-job.test.js >/dev/null 2>&1 && echo pass || echo fail; done | sort | uniq -c
   

        50 pass

   Fifty green runs is not a fix. The same 30 ms version failed twice in 70 runs earlier that day, on a busy machine. Widening a margin lowers a rate, it does not remove a race.

How to read the result

| What you see | What it means | What to do | | --- | --- | --- | | 20 of 20 pass | The rate is below roughly one in twenty, or zero | Loop 50 or 100 times before calling it stable | | A handful of failures, all on one line | One unstable assertion, not a broken suite | Fix or quarantine that test by name | | Failures on different tests each loop | Shared state or ordering, not a timer | Run with --sequence.shuffle and with a single worker to separate the two | | A failing duration just past the asserted limit | A race between real work and a clock | Remove the clock from the assertion, or make the work deterministic | | Tests 3 passed (3) under --retry | A test may have failed and been rescued | Add --reporter=verbose and search the log for retry x | | Red on CI, green on every local loop | The rate depends on machine load | Measure on the busiest machine you have, not the quietest |

Common mistakes

What to check next

• How to check which tests are slowest: the same durations, read for a different reason.
• How to check test coverage: a test that dies early lowers the number it contributed.
• How to check code coverage threshold: a failing test and a missed gate both exit 1.
• How to check if a build is reproducible: the same nondeterminism, one layer down.
• Pull request checklist: where a rerun stops being an answer.

FAQ

What is a flaky test?

A test that returns different results on the same code. It passes and fails with no commit in between, usually because it depends on timing, ordering, shared state, a network call or a random value. The code under test may be correct.

What is test flakiness?

The rate at which a test does that. It is a number, not a label: this one failed 7 of 60 runs at a 25 ms deadline and 2 of 120 at 30 ms.

How do I handle flaky tests once I have found one?

Fix the cause when you can see it, which for a timing race means taking the clock out of the assertion. When you cannot, quarantine the test by name so it stops gating merges, and keep the rate recorded.

Should I use the retry option in CI?

Only with the verbose reporter and a grep for retry x, otherwise retries make the problem invisible. Step 4 also shows them rescuing this test far less often than independence predicts.

How many runs are enough?

Enough to see the rate you care about. Twenty runs miss a one-in-fifty failure about two times in three. Vitest can repeat a test in one process with --repeats, but looping the process is closer to CI, because each run pays the startup cost a timing race is sensitive to.

Verified