When the model won't return valid JSON: a recovery ladder, not a blind retry.

Structured output is a contract, and some models honour it more reliably than others. Anthropic’s tools mode, OpenAI’s strict response_format, and Gemini’s ResponseSchema enforce the shape server-side — they rarely hand back something that fails to parse. Ollama, OpenAI-compatible JSON mode, and any prompt-only endpoint don’t enforce anything: the JSON shape lives in the prompt, and the model is free to ignore it. The post on structured output across four providers described the fallback for when the contract slips — a single retry, then a graceful degrade to plain text.

That fallback had a weakness I want to be honest about: the retry re-sent the byte-identical request. The model that just emitted prose got the exact same prompt again, with no signal about what went wrong. v1.13.0 replaced that blind re-roll with a recovery ladder. This post is about what the ladder does, and why each rung is shaped the way it is.

Three ways the contract breaks

When a prompt-only model fails to produce the findings schema, it fails in one of three recognisable ways:

1. Fenced JSON. The payload is valid findings JSON — but the model wrapped it in a markdown code fence (```json … ```) because it was trained to present code that way. The JSON is perfect; the fence is the only problem.
2. Prose. The model wrote commentary instead of JSON — “Sure, here are the issues I found:” followed by a bulleted list. The schema was ignored outright.
3. Truncation. The model started emitting correct JSON and ran out of room — max_tokens exhaustion mid-object. What came back is a genuine JSON attempt that’s simply cut off.

The old fallback treated all three identically: parse fails, retry the same request, and if that fails too, degrade. But these three failures want three different responses, and the first one shouldn’t cost a retry at all.

Rung 0: salvage the fence for free

The most common failure — fenced-but-valid JSON — is mechanically recoverable with zero provider round-trips. So before anything else, the parser tries a deterministic salvage: if the content is exactly one leading-and-trailing markdown code-fence pair, it strips the fence and re-parses. A response whose body is a fenced json block wrapping a perfectly good {"findings": [ ... ]} object parses on the spot — no retry, no round-trip.

The salvage is deliberately conservative. It unwraps only an exact fence pair — it never goes hunting for a { … } substring buried inside prose. Extracting JSON blobs out of surrounding commentary sounds helpful until you realise it would happily accept a truncated or hallucinated fragment, and it would blur the line between “fenced valid JSON” and “the model ignored the schema.” Non-fenced input passes through byte-for-byte unchanged, which matters for a subtle reason: the review cache key is a hash that includes the response-shaping path, and a salvage that altered clean input would invalidate every existing cache entry. It doesn’t, so they stay valid — and a fenced body that was cached before the upgrade now renders as structured findings on replay.

Rungs 1 and 2: a repair prompt that knows what broke

If salvage doesn’t apply, the parse failure gets classified — empty, prose, truncated JSON, or valid-JSON-wrong-shape — and the single retry sends a repair prompt instead of the identical request. The prompt branches on the failure:

• Prose or schema-ignored → a hard reset: “Your previous response did not parse as valid findings JSON. Output ONLY a single JSON object matching the schema — no prose, no markdown fences, no commentary.”
• Truncated JSON → a completion nudge: the partial output is embedded back into the request with “Your previous response was truncated or malformed; return the COMPLETE corrected JSON object,” and — because truncation is usually a token-ceiling problem, not a comprehension problem — the retry gets a raised max_tokens ceiling.

Telling a model that emitted prose to “complete the JSON” is nonsense; telling a model that got cut off mid-object to “output JSON only, no prose” wastes the retry. Matching the prompt to the failure is the whole point. The retry stays a single, fresh, stateless request — the provider interface is single-shot and semver-frozen, so “continue where you left off” is expressed by embedding the partial output in a new prompt, not by a multi-turn continuation.

The backstop is unchanged

If the repair retry still doesn’t parse, the review degrades exactly as before: the raw response is rendered as plain text, a stderr warning fires, and the cache records markdown-fallback so replays stay quiet. Nothing about the terminal state changed — the ladder just makes it far less likely you reach it. --fail-on still skips its threshold check in the degraded state, for the same reason it always has: there are no structured findings to evaluate.

Making the recovery observable

The one thing the old fallback couldn’t tell you was how often it fired. v1.13.0 adds two optional fields to the --json meta block — retry_count and degrade_reason — emitted only when they’re non-zero, so a clean review’s output is byte-for-byte identical and the schema version stays 1. The --verbose footer surfaces the same signals:

Retries:   1
Degraded:  malformed-json

That’s enough to answer “which of my models actually need the repair path, and why” from the eval harness, instead of guessing. Both fields are live-call-only — a cache hit made no provider call, so it reports neither.

Why not just retry harder

The obvious alternative is retry-N with escalating prompts. I didn’t take it, for the same reason the original design stopped at one retry: a model that fails twice with a corrective prompt is not going to succeed on the fifth attempt, and each attempt is real latency and real tokens on the user’s bill. The ladder’s leverage is in being specific early — salvage the free case, match the prompt to the failure once — not in being persistent. Graceful degrade is the honest ending when a small model just can’t hold the schema; the goal is to reach it rarely, not to thrash before you do.

The asymmetry from the original design still holds: on schema-enforced API providers this whole ladder almost never runs, because their responses are conformant by construction. It earns its keep on the prompt-only providers — Ollama with a small model, an OpenAI-compatible endpoint without strict mode — which is exactly where structured output was always going to be shakiest.

The next post steps away from the LLM’s output entirely, to the checks that don’t need a model at all: the deterministic flaky-test detector that runs as a static pre-pass before a single token is spent.

← all posts