# When the model won't return valid JSON: a recovery ladder, not a blind retry. > Prompt-only models break the JSON contract in three predictable ways — fenced JSON, prose, and truncation. How CommitBrief v1.13.0 recovers each one with fence salvage, a failure-mode-specific repair prompt, and a graceful degrade. Published September 13, 2026 by CommitBrief (https://commitbrief.com) Canonical URL: https://commitbrief.com/blog/when-the-model-wont-return-json/ Tags: json-schema, providers, reliability, engineering --- Structured output is a contract, and some models honour it more reliably than others. Anthropic's tools mode, OpenAI's strict `response_format`, and Gemini's `ResponseSchema` enforce the shape server-side — they rarely hand back something that fails to parse. Ollama, OpenAI-compatible JSON mode, and any prompt-only endpoint don't enforce anything: the JSON shape lives in the prompt, and the model is free to ignore it. [The post on structured output across four providers](/blog/structured-output-across-llms) described the fallback for when the contract slips — a single retry, then a graceful degrade to plain text. That fallback had a weakness I want to be honest about: the retry re-sent the **byte-identical request**. The model that just emitted prose got the exact same prompt again, with no signal about what went wrong. v1.13.0 replaced that blind re-roll with a recovery ladder. This post is about what the ladder does, and why each rung is shaped the way it is. ## Three ways the contract breaks When a prompt-only model fails to produce the findings schema, it fails in one of three recognisable ways: 1. **Fenced JSON.** The payload is *valid* findings JSON — but the model wrapped it in a markdown code fence (` ```json … ``` `) because it was trained to present code that way. The JSON is perfect; the fence is the only problem. 2. **Prose.** The model wrote commentary instead of JSON — "Sure, here are the issues I found:" followed by a bulleted list. The schema was ignored outright. 3. **Truncation.** The model started emitting correct JSON and ran out of room — `max_tokens` exhaustion mid-object. What came back is a genuine JSON attempt that's simply cut off. The old fallback treated all three identically: parse fails, retry the same request, and if that fails too, degrade. But these three failures want three different responses, and the first one shouldn't cost a retry at all. ## Rung 0: salvage the fence for free The most common failure — fenced-but-valid JSON — is mechanically recoverable with zero provider round-trips. So before anything else, the parser tries a deterministic salvage: if the content is exactly one leading-and-trailing markdown code-fence pair, it strips the fence and re-parses. A response whose body is a fenced `json` block wrapping a perfectly good `{"findings": [ ... ]}` object parses on the spot — no retry, no round-trip. The salvage is deliberately conservative. It unwraps **only** an exact fence pair — it never goes hunting for a `{ … }` substring buried inside prose. Extracting JSON blobs out of surrounding commentary sounds helpful until you realise it would happily accept a truncated or hallucinated fragment, and it would blur the line between "fenced valid JSON" and "the model ignored the schema." Non-fenced input passes through byte-for-byte unchanged, which matters for a subtle reason: the review cache key is a hash that includes the response-shaping path, and a salvage that altered clean input would invalidate every existing cache entry. It doesn't, so they stay valid — and a fenced body that was cached before the upgrade now renders as structured findings on replay. ## Rungs 1 and 2: a repair prompt that knows what broke If salvage doesn't apply, the parse failure gets classified — empty, prose, truncated JSON, or valid-JSON-wrong-shape — and the single retry sends a **repair prompt** instead of the identical request. The prompt branches on the failure: - **Prose or schema-ignored** → a hard reset: *"Your previous response did not parse as valid findings JSON. Output ONLY a single JSON object matching the schema — no prose, no markdown fences, no commentary."* - **Truncated JSON** → a completion nudge: the partial output is embedded back into the request with *"Your previous response was truncated or malformed; return the COMPLETE corrected JSON object,"* and — because truncation is usually a token-ceiling problem, not a comprehension problem — the retry gets a raised `max_tokens` ceiling. Telling a model that emitted prose to "complete the JSON" is nonsense; telling a model that got cut off mid-object to "output JSON only, no prose" wastes the retry. Matching the prompt to the failure is the whole point. The retry stays a single, fresh, stateless request — the provider interface is single-shot and semver-frozen, so "continue where you left off" is expressed by embedding the partial output in a new prompt, not by a multi-turn continuation. ## The backstop is unchanged If the repair retry still doesn't parse, the review degrades exactly as before: the raw response is rendered as plain text, a stderr warning fires, and the cache records `markdown-fallback` so replays stay quiet. Nothing about the terminal state changed — the ladder just makes it far less likely you reach it. `--fail-on` still skips its threshold check in the degraded state, for [the same reason it always has](/blog/fail-on-ci-gate): there are no structured findings to evaluate. ## Making the recovery observable The one thing the old fallback couldn't tell you was how often it fired. v1.13.0 adds two optional fields to the `--json` `meta` block — `retry_count` and `degrade_reason` — emitted only when they're non-zero, so a clean review's output is byte-for-byte identical and the schema version stays `1`. The `--verbose` footer surfaces the same signals: ``` Retries: 1 Degraded: malformed-json ``` That's enough to answer "which of my models actually need the repair path, and why" from the eval harness, instead of guessing. Both fields are live-call-only — a cache hit made no provider call, so it reports neither. ## Why not just retry harder The obvious alternative is retry-N with escalating prompts. I didn't take it, for the same reason the original design stopped at one retry: a model that fails twice with a corrective prompt is not going to succeed on the fifth attempt, and each attempt is real latency and real tokens on the user's bill. The ladder's leverage is in being *specific* early — salvage the free case, match the prompt to the failure once — not in being persistent. Graceful degrade is the honest ending when a small model just can't hold the schema; the goal is to reach it rarely, not to thrash before you do. The asymmetry from the original design still holds: on schema-enforced API providers this whole ladder almost never runs, because their responses are conformant by construction. It earns its keep on the prompt-only providers — Ollama with a small model, an OpenAI-compatible endpoint without strict mode — which is exactly where structured output was always going to be shakiest. The next post steps away from the LLM's output entirely, to the checks that don't need a model at all: the deterministic flaky-test detector that runs as a static pre-pass before a single token is spent.