# Flaky-test detector

> A deterministic, provider-free pre-pass that flags timing-dependent and unseeded-random anti-patterns in changed test files before any model call.

CommitBrief docs · v1.x · Reviewing

Canonical URL: https://commitbrief.com/docs/1.x/flaky-tests

---

Some bugs are not in the diff's logic — they are in tests that pass
or fail depending on wall-clock timing, an unseeded RNG, a brittle
selector, or over-mocking. The flaky-test detector is a static,
provider-free pre-pass that catches the highest-precision of these
anti-patterns in your changed test files **before** any model call.
New in v1.7.0; three more rules added in v1.8.0, **opt-in sandbox-rerun
confirmation** in v1.12.0 (ADR-0022), and a bindable test runner for it
in v1.14.0 (ADR-0033).

## What it scans

Only the **added lines** (`+`-prefixed) of files that look like
tests in your current scope — the same diff the reviewer would see,
so the detector never re-flags history already on disk. It runs
entirely locally: no provider call, no extra token cost, no
JSON-schema change.

## What it flags

Five narrow, high-confidence rules:

| Rule | What it catches | Detected forms |
|------|------------------|-----------------|
| `hard-sleep` | Hard-coded sleeps and fixed waits — a test that pauses for a fixed duration to "let things settle" is the classic source of timing flakiness. | `time.Sleep`, `Thread.sleep`, `Task.Delay`, `asyncio.sleep`, `*.waitForTimeout`, a numeric `cy.wait`, `usleep`, `sleep(<n>)` |
| `unseeded-random` | Unseeded randomness — no fixed seed makes a test non-reproducible. | `Math.random`, Python `random.*`, Go `math/rand` |
| `brittle-selector` (JS/TS) | Position-based selectors — a UI/test selector that depends on an element's *position* breaks the moment the DOM shifts. Stable selectors (`data-testid`, role, or text) are **not** flagged. | `:nth-child`, an absolute XPath, `.eq(<n>)` / `.nth(<n>)`, a trailing `[<n>]` index predicate |
| `over-mock` | Excessive mocking — a single test piling on an unusual number of mock setups is a signal the test is brittle and over-coupled to implementation detail. Counted file-scoped, per test function. | — |
| `time-dependency` | Wall-clock-coupled assertions — tied directly to the wall clock **without an injected clock**, so results vary across runs and time zones. Inject a clock / freeze time instead. | `time.Now()`, `Date.now()`, `new Date()` |

The patterns are deliberately conservative: the detector aims for
precision over recall, so a flag is almost always worth a look.

### False-positive caveats

Each rule trades recall for precision, but a few legitimate patterns
can still trip them:

- **`brittle-selector`** can fire on a `[<n>]` index or `.nth(n)` that
  is genuinely the cleanest way to assert against a fixed-order list.
  Switch to a `data-testid`/role/text selector where one exists, or
  add an [inline suppression](/docs/1.x/signal-control#inline-suppression)
  with a reason if the position really is stable.
- **`over-mock`** is a heuristic count, so an integration-style test
  that legitimately wires up many collaborators may cross the line.
- **`time-dependency`** flags wall-clock calls in assertions even when
  you *do* control time another way (a test-only env override, a fixed
  fixture). If the coupling is intentional and safe, suppress it inline.

## Sandbox-rerun confirmation (opt-in)

The rules above *infer* flakiness from anti-patterns. Sandbox-rerun
*confirms* it: it actually **re-runs a flagged test in isolation N
times** and classifies it by the observed pass/fail mix.

| Observed result | Verdict | What CommitBrief does |
|-----------------|---------|------------------------|
| **mixed** pass + fail | **confirmed flaky** | Keep the finding; note the empirical confirmation. |
| **all fail** | **real failure** | Keep it, but relabel — the test is genuinely red, not a flake, so don't quarantine it. |
| **all pass** | **transient / resolved** | Demote the finding to `info`, so a non-reproducing one-off won't trip a commit-stage [`--fail-on`](/docs/1.x/severity). |

The verdict rides the existing finding (its suggestion text and
severity), so there is **no JSON-schema change** — the findings contract
stays v1.

### Double opt-in (v1.14.0)

Arming sandbox-rerun takes **two** settings, and either alone stays
inert (ADR-0033):

1. **How many times** — `--sandbox-rerun[=N]` per run (the bare flag
   uses N=5) or `review.sandbox_rerun: <N>` persistently. `0` = off, the
   default.
2. **What to run** — `review.sandbox_command`, the argv that re-runs a
   single test. Shipped in v1.14.0; before it, the flag was a documented
   no-op.

```yaml
review:
  sandbox_rerun: 5
  sandbox_command: ["go", "test", "-count=1", "-run", "^{{.Test}}$", "./..."]
```

`sandbox_command` is a **list of argv elements, never a shell string**.
Each element is rendered as a Go `text/template` over `{{.File}}` (repo-
relative), `{{.Line}}`, and `{{.Test}}` (the enclosing test function
name), then handed directly to `exec.CommandContext`. No shell is
invoked, so there is no quoting or injection surface.

`commitbrief config set review.sandbox_command` is **rejected** — edit
the YAML by hand. A config surface that can arm code execution gets the
same friction as one that can disable secret scanning.

### What running it actually costs you

- **It executes your repository's code.** The review path never did
  before, so it is never silent about it: a stderr notice names the
  configured command template — the un-rendered argv — once per review,
  before any per-finding rendering.
- **Each attempt has its own 2-minute timeout**, so one hung test costs
  one attempt, not the whole review.
- **It runs against the working tree**, not the staged snapshot a review
  may be scoped to, because that's what the bound command actually
  executes against.
- **It is not cached.** The flaky pre-pass runs *before* the cache
  lookup, so a repeated review of the same diff re-executes the command
  even when the review body itself is served from cache.
- **Test-name resolution is Go-only.** `{{.Test}}` comes from parsing
  `*_test.go` with `go/parser`; every other language returns no name. A
  finding whose test name can't be resolved **skips the rerun** (with a
  stderr warning) and keeps its bare static finding. Python, JS, PHP,
  and Java tests keep full static detection — they just never get
  sandbox-rerun confirmation.

### Never in an agent context

[`commitbrief mcp`](/docs/1.x/mcp-server) and
[`commitbrief guard`](/docs/1.x/policy-gate) **never** run the bound
command — unconditionally, with no user-facing toggle. Both drive the
review through the same internal seam, and an agent host must not
execute repository code unattended. Their flaky findings stay at the
static-only confidence level even when a runner is configured.

## How findings surface

Flaky findings are **first-class** — they merge into the same
structured output as the model's findings, which means they:

- render in the [cards](/docs/1.x/output-formats#cards-default-tty),
  [JSON](/docs/1.x/output-formats#json-schema-v1), and
  [markdown](/docs/1.x/output-formats#markdown) just like any other
  finding;
- count toward [`--fail-on`](/docs/1.x/severity) for CI gating;
- are included by [`--copy`](/docs/1.x/output-formats).

There is no separate flaky-only output mode — they live alongside
correctness, security, and the rest. Messages are localized (en/tr).

## When it runs

On by default for the **API providers** (Anthropic, OpenAI, Gemini,
DeepSeek, Mistral, Cohere, Ollama). It does not apply to the
CLI-tool-backed providers (`claude-cli` / `gemini-cli` / `codex-cli`),
which emit pre-formatted plain text with no structured findings to
merge into.

## Turning it off

```sh
# Per-invocation
commitbrief --staged --no-flaky
```

```yaml
# Per-config (whole binary)
review:
  flaky: false   # default true; --no-flaky is the per-run equivalent
```

```sh
commitbrief config set review.flaky false
```

Disable it if you have a dedicated flake-detection layer in CI, or if
a code base legitimately uses these patterns in a way the detector
cannot distinguish.

## See also

- [Severity and CI gating](/docs/1.x/severity) — `--fail-on` and how
  flaky findings map to exit codes.
- [Output formats](/docs/1.x/output-formats) — where flaky findings
  render.
- [Signal control](/docs/1.x/signal-control) — silence a single
  false-positive flaky finding inline, or baseline a brownfield repo's
  existing ones.
- [Configuration](/docs/1.x/configuration) — `review.flaky`,
  `review.sandbox_rerun`, and `review.sandbox_command` in the config
  reference.