// docs · v1.x

Flaky-test detector

A deterministic, provider-free pre-pass that flags timing-dependent and unseeded-random anti-patterns in changed test files before any model call.

Some bugs are not in the diff’s logic — they are in tests that pass or fail depending on wall-clock timing, an unseeded RNG, a brittle selector, or over-mocking. The flaky-test detector is a static, provider-free pre-pass that catches the highest-precision of these anti-patterns in your changed test files before any model call. New in v1.7.0; three more rules added in v1.8.0, opt-in sandbox-rerun confirmation in v1.12.0 (ADR-0022), and a bindable test runner for it in v1.14.0 (ADR-0033).

What it scans

Only the added lines (+-prefixed) of files that look like tests in your current scope — the same diff the reviewer would see, so the detector never re-flags history already on disk. It runs entirely locally: no provider call, no extra token cost, no JSON-schema change.

What it flags

Five narrow, high-confidence rules:

Rule	What it catches	Detected forms
`hard-sleep`	Hard-coded sleeps and fixed waits — a test that pauses for a fixed duration to “let things settle” is the classic source of timing flakiness.	`time.Sleep`, `Thread.sleep`, `Task.Delay`, `asyncio.sleep`, `*.waitForTimeout`, a numeric `cy.wait`, `usleep`, `sleep(<n>)`
`unseeded-random`	Unseeded randomness — no fixed seed makes a test non-reproducible.	`Math.random`, Python `random.*`, Go `math/rand`
`brittle-selector` (JS/TS)	Position-based selectors — a UI/test selector that depends on an element’s position breaks the moment the DOM shifts. Stable selectors (`data-testid`, role, or text) are not flagged.	`:nth-child`, an absolute XPath, `.eq(<n>)` / `.nth(<n>)`, a trailing `[<n>]` index predicate
`over-mock`	Excessive mocking — a single test piling on an unusual number of mock setups is a signal the test is brittle and over-coupled to implementation detail. Counted file-scoped, per test function.	—
`time-dependency`	Wall-clock-coupled assertions — tied directly to the wall clock without an injected clock, so results vary across runs and time zones. Inject a clock / freeze time instead.	`time.Now()`, `Date.now()`, `new Date()`

The patterns are deliberately conservative: the detector aims for precision over recall, so a flag is almost always worth a look.

False-positive caveats

Each rule trades recall for precision, but a few legitimate patterns can still trip them:

brittle-selector can fire on a [<n>] index or .nth(n) that is genuinely the cleanest way to assert against a fixed-order list. Switch to a data-testid/role/text selector where one exists, or add an inline suppression with a reason if the position really is stable.
over-mock is a heuristic count, so an integration-style test that legitimately wires up many collaborators may cross the line.
time-dependency flags wall-clock calls in assertions even when you do control time another way (a test-only env override, a fixed fixture). If the coupling is intentional and safe, suppress it inline.

Sandbox-rerun confirmation (opt-in)

The rules above infer flakiness from anti-patterns. Sandbox-rerun confirms it: it actually re-runs a flagged test in isolation N times and classifies it by the observed pass/fail mix.

Observed result	Verdict	What CommitBrief does
mixed pass + fail	confirmed flaky	Keep the finding; note the empirical confirmation.
all fail	real failure	Keep it, but relabel — the test is genuinely red, not a flake, so don’t quarantine it.
all pass	transient / resolved	Demote the finding to `info`, so a non-reproducing one-off won’t trip a commit-stage `--fail-on`.

The verdict rides the existing finding (its suggestion text and severity), so there is no JSON-schema change — the findings contract stays v1.

Double opt-in (v1.14.0)

Arming sandbox-rerun takes two settings, and either alone stays inert (ADR-0033):

How many times — --sandbox-rerun[=N] per run (the bare flag uses N=5) or review.sandbox_rerun: <N> persistently. 0 = off, the default.
What to run — review.sandbox_command, the argv that re-runs a single test. Shipped in v1.14.0; before it, the flag was a documented no-op.

review:
  sandbox_rerun: 5
  sandbox_command: ["go", "test", "-count=1", "-run", "^{{.Test}}$", "./..."]

sandbox_command is a list of argv elements, never a shell string. Each element is rendered as a Go text/template over {{.File}} (repo- relative), {{.Line}}, and {{.Test}} (the enclosing test function name), then handed directly to exec.CommandContext. No shell is invoked, so there is no quoting or injection surface.

commitbrief config set review.sandbox_command is rejected — edit the YAML by hand. A config surface that can arm code execution gets the same friction as one that can disable secret scanning.

What running it actually costs you

It executes your repository’s code. The review path never did before, so it is never silent about it: a stderr notice names the configured command template — the un-rendered argv — once per review, before any per-finding rendering.
Each attempt has its own 2-minute timeout, so one hung test costs one attempt, not the whole review.
It runs against the working tree, not the staged snapshot a review may be scoped to, because that’s what the bound command actually executes against.
It is not cached. The flaky pre-pass runs before the cache lookup, so a repeated review of the same diff re-executes the command even when the review body itself is served from cache.
Test-name resolution is Go-only. {{.Test}} comes from parsing *_test.go with go/parser; every other language returns no name. A finding whose test name can’t be resolved skips the rerun (with a stderr warning) and keeps its bare static finding. Python, JS, PHP, and Java tests keep full static detection — they just never get sandbox-rerun confirmation.

Never in an agent context

commitbrief mcp and commitbrief guard never run the bound command — unconditionally, with no user-facing toggle. Both drive the review through the same internal seam, and an agent host must not execute repository code unattended. Their flaky findings stay at the static-only confidence level even when a runner is configured.

How findings surface

Flaky findings are first-class — they merge into the same structured output as the model’s findings, which means they:

render in the cards, JSON, and markdown just like any other finding;
count toward --fail-on for CI gating;
are included by --copy.

There is no separate flaky-only output mode — they live alongside correctness, security, and the rest. Messages are localized (en/tr).

When it runs

On by default for the API providers (Anthropic, OpenAI, Gemini, DeepSeek, Mistral, Cohere, Ollama). It does not apply to the CLI-tool-backed providers (claude-cli / gemini-cli / codex-cli), which emit pre-formatted plain text with no structured findings to merge into.

Turning it off

# Per-invocation
commitbrief --staged --no-flaky

# Per-config (whole binary)
review:
  flaky: false   # default true; --no-flaky is the per-run equivalent

commitbrief config set review.flaky false

Disable it if you have a dedicated flake-detection layer in CI, or if a code base legitimately uses these patterns in a way the detector cannot distinguish.