Proving the review happened: SOC2, audit chains, and the compliance angle of `--json`.

A SOC2 auditor doesn’t ask whether your code is good. They ask whether you followed your stated process for reviewing it. The distinction is everything. If your control says “every production change is peer-reviewed before merge,” the auditor wants the evidence chain that proves it, not an assessment of the reviews’ quality. This is the part of compliance that gets misunderstood most often, and it’s the lens through which I want to close this series.

CommitBrief is not a compliance tool. I want that to be the first thing I say in this post, because the opposite framing would be both inaccurate and irresponsible. What CommitBrief can do is produce artifacts that fit into a compliance evidence pipeline. That distinction is important, and the gap between “evidence pipeline” and “compliance tool” is where this post lives.

What an auditor actually wants

The auditor asks four questions, in roughly this order:

1. Do you have a documented control that says X must happen for production code changes?
2. Can you show that X happened, for a sample of changes drawn from a specific time window?
3. If X happens automatically, can you demonstrate the automation works reliably?
4. If X depends on human judgment, can you show the humans were qualified and the judgment was applied consistently?

LLM-based code review can contribute to question 2 and partially to question 3. It does not contribute to question 4 because an LLM is not a “qualified human” in the auditor’s frame, and they’re right not to treat it as one.

So when I say CommitBrief fits into a compliance pipeline, what I mean specifically is: it produces structured, reproducible artifacts that can serve as part of the evidence for question 2. It is one rung in a ladder, not the ladder itself.

What the JSON output gives you

commitbrief --json --output /tmp/review.json --staged produces a file like this (schema v1, locked at v1.0.0):

{
  "schema": "v1",
  "content": "",
  "findings": [
    {
      "severity": "critical",
      "file": "internal/auth/session.go",
      "line": 142,
      "line_end": 144,
      "title": "SQL fragment built from request input",
      "description": "String concatenation feeds db.Query directly; any value derived from the request URL or body can escape parameterisation here.",
      "suggestion": "Switch to a prepared statement with $1 placeholders; route the user-supplied filter through the existing pgx.NamedArgs path used in queries.go.",
      "language": "go",
      "snippet": "..."
    }
  ],
  "summary": { "total": 1, "by_severity": { "critical": 1 } },
  "meta": {
    "provider": "anthropic",
    "model": "claude-sonnet-4-6",
    "lang": "en",
    "tokens": { "input": 1840, "output": 612, "cached": 1400 },
    "cost_usd": 0.0042,
    "latency_ms": 4123
  }
}

The schema string is "v1" and follows strict semver from v1.0.0 onwards — additive changes (new optional fields) are allowed in v1.x; renames, removals, or type changes require a "v2" bump and a CHANGELOG entry. Severity is one of critical | high | medium | low | info. That stability is the half of the auditor’s question that the JSON output answers directly: here is the finding, in a shape we promised not to break.

The cache key as a reproducibility primitive

The other half — how do you know this exact review actually ran against this exact code? — lives in the local response cache. Every review writes (or replays) an entry at <repo>/.commitbrief/cache/<sha>.json with this shape:

{
  "version": 1,
  "created_at": "2027-06-30T10:14:22Z",
  "ttl": 604800,
  "key": {
    "diff_hash": "sha256:a7f3...",
    "system_prompt_hash": "sha256:c0b4...",
    "provider": "anthropic",
    "model": "claude-sonnet-4-6",
    "lang": "en"
  },
  "result": {
    "content": "<LLM response>",
    "format": "json",
    "tokens": { "input": 1840, "output": 612, "cached": 1400 }
  }
}

The composite cache filename is a SHA-256 of diff + system_prompt + provider + model + lang + schema_version. Six inputs, hashed together. The key.diff_hash and key.system_prompt_hash fields are real per-input SHA-256 digests of their respective inputs (a v1.0.0-rc.1 audit fixed an earlier bug where these fields held a slice of the composite key rather than the actual hashes). That makes each cache file a fingerprint of “this exact review configuration was run against this exact diff.”

For an audit trail, two artifacts with the same diff_hash + system_prompt_hash represent the same review. A cache file with a system_prompt_hash that doesn’t match any of your archived COMMITBRIEF.md versions is suspicious — someone ran the review against rules you don’t have a record of. A cache file with a diff_hash that doesn’t match any commit’s diff is suspicious for the obvious reason.

For a SOC2-style audit, the cache file is the reproducibility primitive; the --json output is the human-readable artifact. Archive both for a complete evidence chain — or archive only the cache file if disk is tight, since the JSON content lives inside result.content already.

What this is not

I want to be explicit about the limits of this framing, because compliance is the area where overclaiming does the most harm.

• An LLM finding is not a remediation. If the review surfaced a blocker and the diff went in unchanged, the artifact is evidence the review happened — not evidence the issue was fixed. Your process has to handle that separately.
• An LLM-only review does not satisfy a “peer review” control. Most SOC2 frameworks require a qualified human reviewer. The LLM artifact can be an additional piece of evidence — “and also, an automated review was run that flagged or did not flag the following items” — but it cannot be the only piece.
• Cache hits look the same as misses in some auditor frames. A cache_hit: true artifact represents a review that was already performed on the same inputs at some earlier point. That’s perfectly valid as evidence, but you may need to walk the auditor through why a cached result is the same as a fresh one. The --verbose “Saved:” label exists partly to make that conversation easier.
• You still need humans. The manifesto made this point in the first post of this series; it returns here as a compliance principle, not just an engineering one.

A practical pattern for evidence-grade pipelines

If you’re trying to wire LLM review into an evidence pipeline today, the pattern that works:

1. Run commitbrief --json --output ./artifacts/review-$(git rev-parse HEAD).json --staged as a required step in your pre-merge workflow. (CI or a git hook, whichever fits your environment.) Pair it with --fail-on=critical if you want the gate to block on critical-severity findings.
2. Archive the corresponding cache file from .commitbrief/cache/ alongside the JSON output. The cache filename is the composite hash; the contents include the per-input diff_hash and system_prompt_hash that make the artifact verifiable.
3. Send both files to immutable storage (S3 with object lock, an append-only audit log, whichever you already have set up).
4. In your control documentation, write down: “Every change to the main branch is reviewed by (a) at least one human reviewer with merge authority, and (b) an automated review tool whose JSON output and cache fingerprint are archived. The diff_hash + system_prompt_hash in the cache file serve as the reproducibility fingerprint.”

That’s four steps. None of them require a “compliance tool.” They require the JSON output to be stable, the cache file’s hashes to be deterministic, and your storage to be trustworthy. CommitBrief provides the first two; your storage layer provides the third.

The point of going to this trouble is that “we have an LLM that helps with review” is a sentence an auditor will accept as a process improvement but reject as a process. “We have an automated review step whose artifacts are archived with reproducible identifiers, supplementing required human review” is a sentence they’ll accept as a process. The difference is what you produce, not just what you do.

Closing the year

This is the fourteenth post in a series that started with a manifesto. The thread running through all of them — pre-commit timing, tool choice, async workflows, provider strategy, and now compliance — is that LLM-based code review is most valuable as the smallest reasonable layer in a stack, not as a layer that replaces other layers.

A year of writing about this has changed my own thinking on it. When I started, I would have called the tool a “second reviewer.” I now think “zeroth reviewer” is the more honest framing, and the framing that does the least damage to the human-review practices teams have built. If this series has made the case for anything, it’s that the question to ask of any tool — review, AI-assisted or otherwise — is not “what does this replace?” but “what does this make room for?”

Thanks for reading along. CommitBrief is open source; if any of these posts shaped how you think about review, the best feedback is the one that lands in the issue tracker or the discussions. The next year of work depends on what you tell me.

---

← all posts