Systems life cycle

What you’ll be able to do

• Apply the software life cycle to a system whose model can change under you
• Know why operate-and-maintain carries more weight for LLM systems
• Plan an evaluation gate that catches behaviour regressions
• Decide when a model upgrade is a deploy and when it is an incident

What you need to know

The one property that makes this different

Plan, build, operate, maintain — the shape is ordinary. One thing is not: traditional software only changes when you deploy it. A Claude application riding a rolling model alias can behave differently tomorrow with no commit, no deploy, and no change on your side at all.

That single property is why this sub-skill exists on the blueprint, and why operate-and-maintain carries disproportionate weight compared to a normal SDLC question.

Planning: decide what “working” means before you build

For conventional software, correctness is mostly binary — the test passes or it doesn't. For an LLM system, output quality is a distribution, so "working" has to be defined as a measurable threshold before you have anything to regress against.

In practice that means writing the evaluation set during planning, not after the first incident. A team that ships without one has no way to answer "did that change make it worse?" — which is the question they will be asked within a month.

Building: prompts are artefacts, not configuration

The build phase treats four things as versioned source: the prompt, the model version, the tool definitions, and the project instructions. Any of the four changing alters behaviour, so any of the four changing should go through review.

Teams routinely version the first and skip the rest, which produces the classic "nothing changed" incident where something very much did.

Operating: the evaluation gate

The named practice is an eval gate — a fixed set of representative inputs with expected characteristics, run before any prompt or model change reaches production. It's a regression test for a probabilistic system.

Maintaining: an upgrade is a deliberate act

Key concept

Your model can change without your deploy. That makes version pinning, an evaluation gate, and treating prompts as reviewed artefacts the operational core of this domain — not optional maturity.

When a scenario says behaviour changed and no code shipped, look for an unpinned reference before you look anywhere else.

Practice scenario

Build exercise — Build the eval gate you don’t have

Intermediate · 35 min

What you’ll learn

• Defining ‘working’ as something measurable for a probabilistic system
• Catching a regression before users do

1. Collect 15–20 real inputs your system has handled, covering the normal case and the awkward ones. For each, write what a good output must contain — not the exact text, but the checkable properties.

   • Why: An eval set is the only thing that lets you answer ‘did that change make it worse?’. Properties beat exact matches because the output is non-deterministic.
   • You should see: A file of inputs paired with assertions like ‘mentions the insurer name’ or ‘returns exactly four sections’.

2. Run the set against your current prompt and record the pass rate. That number is your baseline.

   • Why: Without a baseline, every later comparison is a vibe.
   • You should see: A single percentage. It may be lower than you expect, which is the point.

3. Change one thing in the prompt — a single instruction — re-run, and compare.

   • Why: One change at a time is what makes the comparison mean anything.
   • You should see: A measurable delta. Now you have a gate you could put in CI.

Exam traps

Treating an unexplained behaviour change as a model defect

With a rolling alias in production, it is a configuration failure on your side. Pin the version and upgrade deliberately.

Shipping prompt changes without an evaluation gate

For a probabilistic system, no gate means no way to know whether a change improved or degraded output until users tell you.

Versioning code but not prompts, tools, or model references

All four change behaviour. Versioning only the first produces the “nothing changed” incident.

Defining success only after the system is built

Without an eval set written during planning, there is nothing to regress against later.

Sources

• Claude Developer Platform Docs