# Catching flaky tests before they merge: a deterministic pre-pass, not another prompt. > A hard-coded sleep or an unseeded random seed doesn't need an LLM to spot. CommitBrief's flaky-test detector is a static, zero-token pre-pass — and an optional sandbox-rerun that confirms a flake by actually re-running the test. Published September 18, 2026 by CommitBrief (https://commitbrief.com) Canonical URL: https://commitbrief.com/blog/catching-flaky-tests-before-merge/ Tags: flaky-tests, ci, testing --- Not every problem in a diff needs a language model to find it. [The previous post](/blog/when-the-model-wont-return-json) was about making the LLM's output trustworthy; this one is about the checks that skip the LLM entirely. A `time.Sleep(2 * time.Second)` in a test, a `rand.Intn` with no seed, an assertion on `time.Now()` — these are flaky-test anti-patterns with near-zero false-positive rates, and paying an LLM to notice them is both slower and less reliable than a few regexes that run before a single token is spent. That's what CommitBrief's flaky-test detector is: a deterministic static pre-pass over the added lines of changed test files. It emits the same `Finding` shape the LLM review produces, so everything downstream — the cards renderer, `--json`, `--fail-on` — treats a flaky finding exactly like a review finding. It's on by default; `--no-flaky` skips it for a run, and `review.flaky: false` disables it in config. ## What it looks for The detector scans **added** lines in files that look like tests, matching a small set of high-precision anti-patterns: - **Hard-coded sleeps** — `time.Sleep`, `Thread.sleep`, `await asyncio.sleep(…)` with a literal duration, used to "wait for" async work. The classic source of tests that pass on a fast machine and fail in CI. - **Unseeded randomness** — `rand.Intn`, `Math.random()`, `random.choice` feeding test data or control flow with no fixed seed, so the test exercises a different path every run. - **Time dependencies** — assertions built on `time.Now()`, `Date.now()`, or the system clock, which fail at midnight, across a DST boundary, or in a different timezone. - **Brittle selectors** — UI/E2E locators pinned to auto-generated class names or absolute XPath indices that shift the moment the DOM changes. - **Over-mocking** — a test so thoroughly mocked that it asserts against its own mocks and validates nothing about the real code path. Each match becomes a finding with a severity, a line anchor, and a suggestion — "seed the RNG," "inject a clock," "wait on a condition, not a duration." The point isn't to be clever; it's to be **precise**. A flaky-test detector that cries wolf gets turned off in a week, so the rule set is deliberately narrow: high confidence, low recall, no heuristics that would fire on legitimate code. ## Why deterministic, not an LLM prompt The tempting alternative was to fold flaky-test awareness into the review prompt — "also flag flaky-test anti-patterns." I chose the static pre-pass instead, for three reasons. **It's free.** The scan runs locally over the diff before the provider call. No tokens, no latency, no dependence on which model you're using or how it's feeling today. **It's deterministic.** The same diff produces the same findings every run. A prompt-augmented approach would flag a `time.Sleep` sometimes and miss it other times, depending on sampling — the worst property for something you want to wire into a CI gate. **It composes.** Because a flaky finding is a normal `Finding`, it flows into `--fail-on` and `--json` with no special casing. `commitbrief --staged --fail-on high` blocks a merge on a critical flaky pattern the same way it blocks on a critical review finding. The trade-off is honest: a static pre-pass only catches patterns someone encoded a rule for. It won't reason about a subtle race that has no textual tell. That's fine — it's a floor, not a ceiling. The LLM review still runs on the same diff and can catch the things a regex can't. ## Confirming a flake by actually running it A static pattern *infers* flakiness; it can't *prove* it. A `time.Sleep` might be load-bearing in a way the pattern can't see. So there's an opt-in second stage that raises confidence by doing the obvious thing — running the flagged test in isolation, several times, and watching what happens: ```sh # Re-run each statically-flagged test up to 5 times in isolation. commitbrief --staged --sandbox-rerun # Or set the rerun count explicitly. commitbrief --staged --sandbox-rerun=10 ``` The verdict comes from the observed pass/fail mix: - **Mixed pass *and* fail** → confirmed **flaky**. The test is genuinely non-deterministic. - **All fails** → a **real failure**. It's red for a reason — not a flake to quarantine, an actual bug. - **All passes** → **transient**. The flake didn't reproduce this time, so the finding is demoted to `info` rather than dropped, because "didn't reproduce in five runs" isn't "definitely fine." There's an early exit built in — a genuinely flaky test usually shows a mixed result within the first two runs, so the loop stops as soon as it has proof rather than always running the full N. Sandbox-rerun is off by default and sits behind an **executor seam**: the core ships the orchestration — the rerun loop, the classification, the early exit — but not a language-specific test runner. You (or an integration) bind a `func(ctx, testID) (passed bool, err error)` that knows how to run one test in your project. That keeps the detector's core pure and testable with a fake, and keeps CommitBrief out of the business of embedding a runner for every test framework in existence. ## Where it sits in the pipeline The flaky pre-pass runs before the provider call, and its findings are merged with the LLM's — with one rule: if the detector and the model both flag the same line, the model's richer finding wins and the duplicate is dropped. So you don't get two findings for one `time.Sleep`; you get the better of the two. The net effect is a review that opens with the cheap, certain, deterministic checks and layers the model's judgement on top. The next post stays on the theme of feeding the review things a plain diff can't tell it — this time, your project's own declared architecture, read straight from a sibling tool's config so the model can flag a diff that crosses a boundary it was never told about.