v1.13 → v1.15: replacing inference with evidence

Three releases went out in three weeks. From the outside they don’t look like they belong to the same arc: v1.13.0 changed how a malformed JSON response is recovered, v1.14.0 added a single config key, and v1.15.0 shipped four new pieces of command surface. Different files, different subsystems, no shared code.

They are the same idea, applied three times. Every one of them takes something CommitBrief was inferring and replaces it with something it can show you — and, where the answer never needed a language model at all, stops paying for one.

v1.13.0 — stop retrying the same request and hoping

The findings schema is a contract. Anthropic’s tools mode, OpenAI’s strict response_format, and Gemini’s ResponseSchema enforce it server-side. Ollama, OpenAI-compatible JSON mode, and any prompt-only endpoint don’t: the shape lives in the prompt, and the model is free to ignore it.

The old fallback for that case was one retry, then a graceful degrade to plain text. The weakness was the retry itself — it re-sent the byte-identical request. A model that just emitted prose got the exact same prompt back, with no signal about what went wrong. That isn’t a recovery strategy; it’s a re-roll.

v1.13.0 replaced it with a ladder. The most common failure — valid findings JSON wrapped in a markdown code fence — is now stripped and re-parsed before anything else, at zero round-trips, so it costs no retry at all. If that doesn’t apply, the parse error is classified (empty / prose / truncated / wrong shape) and the single retry sends a failure-mode-specific repair prompt: a hard “JSON only” reset for prose, or a “complete the JSON” nudge with the partial output embedded and a raised max_tokens for a truncated attempt. Telling a model that emitted prose to “complete the JSON” is nonsense; matching the prompt to the failure is the whole point.

The part that matters for this arc is the second half of that release: meta.retry_count and meta.degrade_reason in the --json output, plus two --verbose footer lines. Before them, nobody could answer “how often does this actually fire, and on which models?” The recovery got better and auditable in the same release. The v1.13.0 notes have the details; a full walkthrough of the ladder is queued for a later post.

v1.14.0 — a detector that infers, and a runner that confirms

The flaky-test detector has always been honest about what it is: a static pre-pass that infers flakiness from anti-patterns. A time.Sleep in a test is evidence of a timing dependency, not proof of one.

v1.12.0 shipped the machinery to close that gap — re-run a flagged test N times in isolation, classify by the observed pass/fail mix — behind an executor seam with nothing bound to it. It was a documented no-op, which is a strange thing to ship, and I said so at the time. v1.14.0 binds it:

review:
  sandbox_rerun: 5
  sandbox_command: ["go", "test", "-count=1", "-run", "^{{.Test}}$", "./..."]

Now a mixed pass+fail confirms the test is flaky, all fails means it’s genuinely red rather than a flake to quarantine, and all passes demotes the finding to info. Inference became measurement.

Executing repository code from a review path is a real boundary to cross, so the design is deliberately grudging about it:

• It’s a list of argv elements, never a shell string. Each element is a Go text/template over {{.File}}, {{.Line}}, {{.Test}}, handed straight to exec.CommandContext. No shell means no quoting rules to get wrong and no injection surface to reason about.
• Double opt-in. A positive sandbox_rerun and a non-empty sandbox_command. Either alone stays inert, so the default path is byte-identical to what shipped before.
• config set review.sandbox_command is rejected. Hand-edit only — the same friction guard.secret_patterns gets. A key that can arm code execution shouldn’t be settable from a one-liner in someone’s shell history.
• It announces itself. A stderr notice names the configured command template, un-rendered, once per review, before any per-finding work. The review path had never executed code; it is not going to start doing it quietly.
• mcp and guard never run it — unconditionally, no toggle. An agent host must not execute repository code unattended, so their flaky findings stay at static-only confidence even when a runner is configured.

One honest limit: {{.Test}} resolution is Go-only. It parses *_test.go with go/parser. Three rounds of a hand-rolled multi-language scanner kept producing confident wrong test names, which is worse than not confirming at all, so it now returns nothing for every other language and the finding skips the rerun with a warning. Python, JS, PHP, and Java tests keep full static detection; they just don’t get empirical confirmation yet.

v1.15.0 — the questions that never needed a model

Then the same idea ran off the end of the LLM entirely.

Reviewing a set of commits, and seeing which ones

Commit filters — --author, --committer, --start-date, --end-date, --text — select a set of commits rather than one diff. git diff has no author or date options; those belong to git log. So setting any of them switches diff acquisition to a commit walk: pick the matching commits, concatenate their patches.

commitbrief --author alice --start-date 2026-06-01 --dir internal

Filter kinds are AND’d, values within a kind are OR’d. --end-date is inclusive of the day you named, because git’s bare --until stops at that day’s midnight and silently drops it — a footgun I wasn’t going to re-export.

And here is where the arc shows: the moment you can select a non-contiguous set of commits from anywhere in history, the only feedback you get is a number.

$ commitbrief dry-run --author alice --start-date 2026-06-01
Commits (matched): 12

Which twelve? For a filter whose entire job is deciding what gets reviewed, “trust me, twelve” is weak feedback, and a wrong filter quietly reviews the wrong code — then bills you for it. So commitbrief map draws the DAG with matches highlighted and everything else dimmed as context:

● a1b2c3d  (main) feat: add commit filters        alice  2h
│╲
│ ○ e4f5a6b  (payments/stripe) chore: bump sdk    bob    3d
│ ○ b7c8d9e  fix: rounding                        bob    3d
│╱
● f0a1b2c  docs: readme                           carol  1w

● matches the filter   ○ context

The non-matching commits stay in the picture on purpose. A graph of only the matches is a list, and its lanes would be meaningless. map is deterministic, always exits 0, and rejects --json, --fail-on, and --min-severity rather than pretending to be a gate. It’s a viewer — the cheapest possible way to check a filter before paying for a review.

The gate is not the audit

The pre-send secret scanner has always been a gate. It sees exactly one thing: the added lines of the diff about to leave your machine. That’s the right shape for stopping a leak in flight, and the wrong shape for the two questions people actually ask:

• Is there a key sitting in my working tree right now?
• Did anyone ever commit one?

Neither is answerable from one diff. And the second one matters more than it looks, because removing a key from the working tree doesn’t remove it from the history — it stays reachable in every clone, every fork, and every CI cache that already pulled it.

commitbrief leaks answers both, with the same eight built-in patterns plus your guard.secret_patterns, and no provider call, no cache, no cost:

commitbrief leaks                      # tracked files + the last 200 commits
commitbrief leaks --no-history         # working tree only, fast
commitbrief leaks --json | commitbrief guard --from-json -

Two halves, both on by default, each with its own off-switch — so a positional range narrows the history scan without silently disabling the tree scan. The working-tree half reads every tracked file whole. Untracked, gitignored files are deliberately out of scope: an untracked .env is exactly where a secret is supposed to live, and it cannot leak through git. The history half reads only the added lines of the selected commits, attributed to commit, author, and date — which is both how a removed-but-committed key gets found, and what tells you whose credential needs rotating.

It exits 1 on any hit, so it gates CI with no configuration. It emits the existing schema v1 with meta.provider: "builtin", so the policy gate consumes it with no new plumbing — a separate scan schema would have bought nothing and broken that pipeline.

And it never prints the matched text. A finding carries a file, a line, and the pattern names. The scanner reads whole files; its own report must not become a second copy of the secret in your scrollback, your CI log, or the pastebin you were about to send it to.

Two limits stated in the docs rather than buried: it honors the ignore layers, so a key committed into vendor/** is not reported, and it’s regex-only — a high-entropy blob with no recognizable prefix is invisible to it. This is a targeted check, not a general-purpose secret scanner, and a clean report means nothing if you don’t know what was looked at. That’s why coverage — files scanned, commits scanned, files skipped, whether the walk was truncated — is always printed alongside the result.

The smaller pieces

• --exclude-file / --exclude-dir. The inverse of --file / --dir, same matching rules, applied after them so an exclusion always wins: --dir internal --exclude-dir internal/cli.
• Filters over MCP. The review tool gained all of the path and commit filters. The path pair had been CLI-only in practice — the MCP seam resets global flag state, so an agent host had no way to narrow a review by path at all.
• meta.filtered_commits. Optional and omitempty, so the schema stays 1 — how many commits’ patches make up the reviewed diff. dry-run grew matching Commits (walked) / Commits (matched) lines. Same instinct as v1.13’s retry_count: if the tool made a decision on your behalf, it should be able to tell you what the decision was.
• A bug fix worth naming. remote pr only honored --file / --dir on the --no-post path. A narrowed run that actually commented on GitHub reviewed a different file set than the same command with --no-post. That’s the worst kind of bug — the preview lied about the real thing.

And one that’s just overdue

commitbrief upgrade detects how the running binary was installed and does the right thing for it. Homebrew, Scoop, and go install are delegated to their own package manager, because overwriting a manager-owned binary desynchronizes its bookkeeping and its next upgrade either conflicts or silently reverts your swap. Only a manual install gets replaced in place, after its SHA-256 is checked against the release checksums.txt.

An unwritable target aborts before anything is downloaded and prints the exact command to run. CommitBrief never invokes sudo itself — whether to elevate is your call, not a decision a CLI should make for you while holding a downloaded binary.

Two things I’d rather say than let you discover: checksums.txt is unsigned, so the trust anchor is TLS to GitHub. That defends against a corrupted or truncated download, not against a compromised release — anyone who could publish a malicious release could publish a matching checksum file next to it. Artifact signing is future work. And this is the only network request CommitBrief makes on its own behalf, only when you run this command: no background check, no post-review nudge, no config key that would turn one on.

The through-line

Every item above is the same move.

A retry that carries the reason it’s retrying. A flaky finding that was measured instead of guessed. A filter you can look at before it spends your money. A secret scan that reads what’s actually in your repository instead of what’s in one diff. A meta field for every decision made on your behalf.

There’s a second thread underneath it, which is that a surprising amount of what a “code review tool” does doesn’t need a language model. leaks is regex over files and commits. map is lane assignment over a DAG — a pure function in a new internal/graph package, no new dependencies. Both are deterministic, both are free, and both answer questions that a model would have answered less reliably and charged for. The model is for judgment. Everything else should be arithmetic.

Next up, the roadmap points back at the review itself — specifically at what a finding can prove about the code it’s pointing at, now that the pipeline is allowed to run something.

← all posts