Output handling

What you’ll be able to do

Validate model output against a schema rather than parsing it directly
Build a corrective retry that feeds the error back
Apply skepticism toward confident but wrong output
Design output contracts that make deviations loud

What you need to know

Never call a parser on raw model output

Models occasionally wrap JSON in a sentence of prose, omit a field, add one you didn't ask for, or invent an enum value. Not often — often enough that production code has to expect it.

before — crashes on the first deviation

data = json.loads(response.content[0].text) save_to_db(data)

after — validated, with corrective retry

for attempt in range(MAX_RETRIES): raw = extract_json_block(response.content[0].text) try: data = OutputSchema.model_validate_json(raw) break except ValidationError as err: response = client.messages.create( model=MODEL, max_tokens=1024, messages=messages + [ {"role":"assistant","content":raw}, {"role":"user","content": f"That failed validation: {err}. " f"Return only valid JSON matching the schema."}]) else: raise OutputContractError("schema never satisfied")

Three things make that work: extract the JSON block rather than assuming the whole response is JSON, validate against a schema rather than trusting a successful parse, and feed the specific error back so the retry has something to correct.

Skepticism toward confident output

The blueprint's own phrasing, and worth internalising exactly. A wrong answer from a language model is fluent, well-formatted, appropriately hedged, and carries no signal that it's wrong. There is no confidence tell to watch for.

That's why validation belongs in code rather than in a reviewer's judgment. "We'll notice if it's wrong" is not a control — the failure mode is specifically that you won't.

Design the contract so deviations are loud

A field typed as status: string accepts "probably active" and passes it downstream. The same field as an enum of three permitted values rejects it immediately and you find out at the boundary.

General rule: prefer required fields over optional, enums over free strings, and explicit nulls over absent keys. Every loosening you allow is a class of silent wrong value you've agreed to accept.

Where validation failure belongs

ScenarioA claims pipeline extracts a coverage decision from each document. One in 300 responses comes back with a coverage status the schema never defined. The pipeline has no validation, so the value writes to the database, and downstream reporting treats it as a valid category for two months.

The model did something expected and occasionally imperfect. The failure is entirely in the missing contract — and note the cost: with validation, this is a logged retry. Without it, it's two months of corrupted reporting nobody noticed.

Key concept

Extract, validate against a schema, and retry with the error fed back. A wrong answer is fluent and carries no warning, so the check has to live in code — and a tight contract turns silent wrong values into loud failures.

Practice scenario

ScenarioOne response in 300 contains a coverage status your schema never defined. There's no validation, so it writes straight to the database.

Work it through, then open this

The model did something expected and occasionally imperfect; the failure is the missing contract. With schema validation this is a logged retry. Without it, it’s months of corrupted reporting nobody noticed — and note that a successful parse wouldn’t have caught it either.

Build exercise — Break your own parser on purpose

Intermediate · 30 min

What you’ll learn

Validating against a schema rather than trusting a parse
Feeding the error back so a retry can correct

Find where you parse model output and run it against a response that wraps valid JSON in a sentence of prose.
- Why: Models occasionally do this. Code that assumes the whole response is JSON breaks on the first occurrence.
- You should see: A crash, or extraction logic that already handles it.
Now feed it structurally valid JSON containing a field value your schema never defined — an invented enum value.
- Why: This is the dangerous case: a successful parse producing a plausible wrong value that flows downstream unnoticed.
- You should see: Either a validation error at the boundary, or a bad value quietly written to your database.
Add a retry that includes the specific validation error in the follow-up message, and cap the attempts.
- Why: Retrying an identical request reproduces the identical failure. The error message is what gives the retry something to correct.
- You should see: A second attempt that fixes the deviation, and a hard stop rather than an infinite loop.

Exam traps

Calling a JSON parser directly on the response text

A successful parse is not a valid document, and prose around the JSON breaks it entirely.

Retrying without feeding the validation error back

An identical request produces an identical class of failure. The error message is what gives the retry something to correct.

Relying on human review to catch wrong output

Wrong answers are fluent and confident by construction. There is no tell to notice.

Typing constrained fields as free-form strings

It converts an immediate validation error into a plausible wrong value that propagates silently.

Sources

Claude Developer Platform Docs

What you'll be able to do