What you’ll be able to do
- Identify what to log for a production agent beyond just final output
- Use stop_reason distribution and loop iteration counts as health signals
- Design a lightweight eval loop that runs before deployment, not after complaints
- Distinguish ad-hoc manual testing from a repeatable evaluation process
What you need to know
Log the loop, not just the answer
A production agent that only logs its final output is observable the way a stack trace with no line numbers is debuggable — technically present, practically useless when something goes wrong. What actually needs logging for an agentic system:
- Every tool call and its result — including errors, since a tool that's failing silently and being retried into eventual success looks identical to a healthy system if only the final answer is recorded.
- stop_reason on every model turn — a distribution of stop reasons over time is a health signal (see below), not just a per-call detail.
- Loop iteration count per session — how many turns it took to finish, so a session that took 40 turns instead of the usual 5 is visible as an anomaly, not buried in an aggregate latency number.
- Token spend per session — input, output, and cache read/write separately, since a cost regression can hide behind an otherwise-normal-looking total.
None of this is exotic — it's the same instinct as logging request IDs and status codes for a web service, applied to a system whose "requests" are multi-step and whose "status codes" are stop reasons and tool outcomes instead of HTTP codes.
stop_reason distribution is a health dashboard, not a per-call detail
A single max_tokens stop is unremarkable — some answers are long. A rising rate of max_tokens stops across sessions is a different thing entirely: it usually means either the task genuinely got harder (worth knowing) or an upstream prompt change silently pushed the model toward longer answers than the allocated budget expects (worth fixing before users see truncated responses).
The same logic applies to a rising rate of refusal stops (something about recent inputs or a prompt change is triggering more declines) or unexpected tool_use loops that don't terminate as expected. None of these are visible from looking at any single call. They're only visible as a distribution, tracked over time, which is the difference between logging and observability.
An eval loop beats a spot-check, because it’s repeatable
Manually trying a prompt change against three examples before shipping it feels like testing, and it catches the most obvious regressions. It doesn't catch a regression on the fourth case nobody happened to try, and it produces no artifact anyone else can rerun later. A lightweight eval loop fixes both:
- A fixed eval set — a set of representative inputs, including edge cases and known-hard examples, not just happy-path ones. This is the artifact a spot-check never produces.
- A baseline — run the eval set against the current production behavior before changing anything, so "better" or "worse" after a change means something concrete.
- Automated or semi-automated scoring — even a simple pass/fail check against expected properties (did it call the right tool, did the output match a schema) catches more than intuition does, and it catches the same thing every time, which a human reviewer with fatigue doesn't.
This doesn't have to be sophisticated to be valuable. The gap the exam cares about is repeatable versus not — a fixed eval set run before every change beats an ad-hoc check of whatever examples happened to be top of mind that day, even if both are done by the same person with the same judgment.
Key concept
Log the loop — tool calls, stop reasons, iteration counts, token spend — not just the final answer. A fixed eval set run before every change beats a spot-check of whatever examples were top of mind.
When a scenario describes a regression that "should have been caught" before release, or a production issue nobody noticed until users complained, the missing piece is almost always one of these two: nothing was logging the intermediate signal that would have shown it, or nothing was evaluating changes against a repeatable baseline before shipping.
Practice scenario
Work it through, then open this
Two separate gaps compounded. The wording change likely nudged the model toward longer typical answers without anyone connecting that to the fixed output-token budget — invisible without stop_reason distribution being tracked, since no single call proves anything. And the change went out without an eval run against a baseline, so there was no repeatable check that would have surfaced a rising max_tokens rate before users did. Fix both: track stop_reason distribution as a standing metric, and require an eval-set run against baseline before a prompt change ships.
Build exercise — Build a minimal eval loop
Advanced · 35 min
What you’ll learn
- Choosing what to log beyond final output for an agentic system
- Building a fixed eval set that includes edge cases, not just happy paths
- Comparing a change against a baseline instead of judging it in isolation
-
For an existing or planned agent, list what’s currently logged and what’s missing from: tool calls and results, stop_reason per turn, loop iteration count, token spend by category.
- Why: Most systems log final output first and everything else later, if ever — this makes the gap concrete.
- You should see: At least one signal from that list currently invisible in your logs.
-
Build a fixed eval set of 8–10 inputs for one agent behavior, deliberately including at least two edge cases or known-hard examples, not just typical ones.
- Why: A happy-path-only eval set catches nothing a spot-check wouldn’t have caught anyway.
- You should see: A reusable list you could hand to someone else and get the same test run.
-
Run that eval set against current behavior to record a baseline, then simulate a small prompt change and rerun it.
- Why: “Better” or “worse” only means something relative to a baseline captured before the change.
- You should see: A concrete before/after comparison, not an impression of whether the change felt fine.
Exam traps
Logging only final output and losing every intermediate decision the agent made
Makes debugging a failure after the fact nearly impossible — there’s no record of what the agent actually did along the way.
Treating a spike in max_tokens stop_reason as normal instead of a signal
A rising rate over time usually points to a real cause — a harder task mix or an upstream prompt change — worth catching before users see truncated output.
Relying on manual spot-checks as the only quality gate before a release
Catches obvious regressions on whatever examples were tried; produces no repeatable artifact and misses everything not tried.
Building an eval set only from happy-path examples
Confirms the system works when nothing is wrong, which was never in question — edge cases are where regressions actually hide.
Not tracking loop iteration counts, so a runaway agent looks like normal load
A session that takes 8× the usual number of turns is a distinct signal from “traffic is up,” and it’s invisible without per-session tracking.
Evaluating a prompt change without a baseline to compare it against
“Seems fine” isn’t a measurement. A baseline run before the change is what makes “better” or “worse” mean something concrete.