What you’ll be able to do
- Validate output against a schema before it reaches a downstream system with side effects
- Guard against partial output by checking stop_reason first
- Design a repair-or-reject policy sized to the stakes of each field
What you need to know
Three separate checks, not one
"Is this output usable" is really three independent questions, and architecture that collapses them into one check misses failures:
- Is it complete? Check
stop_reasonbefore anything else. Amax_tokensstop means the response was cut off mid-generation — it may look like valid JSON right up to the point it was truncated and still be missing its closing brace. - Is it well-formed? Schema validation — correct types, required fields present, enum values within range.
- Is it correct? Business-logic validation — a well-formed, complete response can still recommend a refund on a non-refundable order. Syntax validation never catches this.
result = validate_schema(response) # types, required fields validate_business_rules(result) # e.g. refund eligibility
downstream_system.apply(result)
Missing fields deserve a decision, not a default
A downstream system that silently substitutes a default value for a missing field — an empty string, a zero, a "false" — hides a real failure behind output that looks complete. Flag a missing required field as a validation failure and route it through the repair-or-reject decision explicitly, rather than papering over it with a value nobody asked for.
Size the response to the stakes of the field
Not every field deserves the same validation rigor. A cosmetic display field with a syntax error is a candidate for a cheap repair turn. A field controlling a financial transaction or a data deletion deserves reject-and-escalate on any validation failure — the cost of a false positive is asymmetric, and repairing your way to a plausible-looking value is the wrong instinct when the downside of being wrong is high.
Silent drift is worse than a loud failure
Validation that never logs its failures can't be improved and can't reveal a slow drift — a prompt or model change that gradually increases the malformed-output rate from 1% to 8% looks fine at a glance if nobody's counting. Log every validation failure with enough context to distinguish "one-off fluke" from "something changed."
Key concept
Completeness, well-formedness, and correctness are three different checks — stop_reason answers the first, schema validation the second, and business-logic validation the third. Skipping any one of them lets a specific class of bad output through.
When a scenario describes output that "looked fine but caused a downstream problem," the missing check is almost always business-logic validation, not syntax.
Practice scenario
Work it through, then open this
The pipeline was checking that the response was well-formed — the tool call matched its schema — but never checking business rules. A digital-good exclusion is a business-logic constraint, not a type or required-field constraint, so schema validation alone would never catch it. The fix adds a business-rule validation pass after schema validation and before the call reaches the payment system, with a reject-and-escalate policy for a financial action rather than a silent pass-through.
Build exercise — Add all three validation layers to a pipeline
Intermediate · 25 min
What you’ll learn
- Separating completeness, schema, and business-rule validation into distinct checks
- Deciding which fields deserve repair versus reject-and-escalate
- Logging validation failures usefully
-
Take a pipeline that currently does one validation pass and split it into a stop_reason check, a schema check, and a business-rule check, in that order.
- Why: Each layer catches a failure the others structurally cannot.
- You should see: At least one of the three layers catching something the current single pass would have missed.
-
For each field in the schema, decide whether a validation failure should trigger repair, reject-and-retry, or reject-and-escalate.
- Why: Treating every field identically either wastes retries on low-stakes slips or risks coercing a high-stakes field toward a plausible-but-wrong value.
- You should see: At least one field reclassified from “always repair” to “always escalate” once its real stakes are considered.
-
Add logging for every validation failure, including which layer caught it.
- Why: Without this, a gradual increase in the failure rate is invisible until it’s a production incident.
- You should see: A log line per failure, distinguishing a completeness failure from a schema failure from a business-rule failure.
Exam traps
Parsing output without checking stop_reason first
A max_tokens truncation can look like valid, complete output right up to where it was cut off.
Assuming a well-formatted response is also a correct one
Schema validation checks shape, not business rules. Both are required; neither substitutes for the other.
Passing unvalidated output directly into a downstream system with side effects
Especially dangerous for actions with real-world consequences — a refund, a deletion, a message sent to a customer.
Silently defaulting a missing field instead of flagging it
Hides a real failure behind output that looks complete, and the eventual downstream problem is much harder to trace back.
Building validation that only checks syntax, not the business-logic constraints that matter
A response can be perfectly well-formed and still violate a rule that matters — schema validation alone will never catch it.
Not logging validation failures, so drift goes unnoticed until it’s a production incident
A slow increase in the failure rate is invisible without logging, and by the time it’s noticed it’s usually already caused damage.