Evals that check the end state, not the text

A transcript is the model's account of its own work, and an account can be wrong. A tool call fails and the model still writes "Done, the reminder is on its way". A retry writes the same row twice and the summary mentions one. If you grade the summary, you're grading how confident the model sounds.

State is what's left once the run is over: rows in the database, messages in the queue, files on disk. Anthropic's guide to agent evals makes the same split. It calls the full record of a run the transcript and the final state of the environment the outcome, and its example is a flight-booking agent that says the flight is booked while the real question is whether a reservation exists in the database.

For an agent that only returns text, such as a search assistant, grading text is fine. For an agent with side effects it isn't enough. An invoicing agent that creates reminders has passed when one correct reminder exists, no second one exists, and no other customer's record changed. You can't read any of that from a transcript.

Three questions about state, and sometimes a fourth about text.

What must exist. The intended change, written as things you can count: one reminder row for this invoice and this step, one queued email for that reminder, one file at this path. List them before the run.

What must not exist. The copies. No second reminder for the same invoice and step. No second email in the queue. Duplicates are the failure I'd design against first, because retries, repeated tool calls and re-run steps all produce them, and the transcript rarely admits it.

What must be unchanged. The boundary. Other customers, other invoices, settings tables, folders the task has no business in. Name a protected set and confirm after the run that nothing in it moved. This is the check that's easiest to skip. An agent that fixes one record and quietly rewrites a hundred others has not succeeded, and only this check will tell you.

Whether the text is acceptable, where someone reads it. A reminder email has a tone. A log line doesn't. For the email I'd use a short rubric: correct recipient, correct invoice number and amount, polite, brief, no invented details. An invented payment link or deadline fails the task even if every count is right.

Fixture. Start from a known state. Seed a test database with a few rows under obviously fake customer names. Fix the dates your logic depends on, so "overdue" means the same thing on every run. Create the folders the agent may write to. Then take a snapshot of every table and folder the agent could reach, including the ones it shouldn't touch.

Run. One task, started the way production starts it, with the same tools the agent has in production. Don't steer it halfway through and don't retry on its behalf. If it asks for confirmation, answer the way your production policy would.

Assert. Compare the state after the run with the snapshot plus the intended change. Give every check a name and record pass or fail for each one, so a failed run tells you which check broke.

Two rules for fixtures. Keep them small, because a suite that takes ten minutes gets skipped, and then prompt changes go out untested. And give every trial its own fresh copy. State left over from the previous trial produces false passes and false fails, and it takes a long time to work out which is which.

Use the real write path. If the eval swaps the database writes for a stub that always succeeds, the duplicates and scope breaks you're hunting can't happen. Point the real tools at test data instead. Where a tool would reach a real person, such as an email sender, point it at a queue that never delivers.

This is a hypothetical invoicing agent. Its task: for one overdue invoice, create one payment reminder and queue one email to the customer. It must never create two reminders for the same invoice and step, never queue the same email twice, and never touch another customer.

The fixture has two customers. Customer A has one overdue invoice at reminder step 1. Customer B has an unrelated invoice and exists only to prove the agent left it alone. The agent may read both. It may write only the reminder and the queued email for customer A.

After the run, the eval checks four groups of things. One reminder row exists for A's invoice at step 1, and one queued email points to it. No second reminder and no second queued email exist. Nothing was sent, because the task ends with the email in the queue. Everything outside the reminders table and the email queue is identical to the snapshot.

Then one extra trial that earns its place: run the send step twice in a row, as a retry would. There should still be one email. If there are two, no prompt will fix it. The write path needs a guard, such as a unique constraint on invoice and step, or an idempotency key on the send.

The email text gets the short rubric from above, and only that.

In code, the state checks come out roughly like this. It's an illustration of the shape and the helper names are made up.

before = snapshot(db)
run_agent(task="remind", invoice="A-001")
after = snapshot(db)

assert count(after.reminders, invoice="A-001", step=1) == 1
assert count(after.outbox, invoice="A-001", status="queued") == 1
assert count(after.outbox, status="sent") == 0
assert diff(before, after, ignore=["reminders", "outbox"]) == {}

One passing run tells you the agent can do the task. It doesn't tell you how often. The model's output varies from run to run, tools answer in a different order, a timeout sends the run down the retry path. So each task gets run several times, each from a fresh fixture, and each trial ends in one verdict: pass only if every state check passed and the text check passed where it applied.

The result is a pass rate per task. Passed trials over total trials, one number per task. Don't blend tasks into a single score, or a weak task hides behind three strong ones.

How strict should you be? Anthropic's guide describes two ways to read repeated trials. pass@k asks whether at least one of k attempts succeeded. pass^k asks whether all k did. For anything a customer sees, the second is the honest one, and the arithmetic is worth doing. If each trial passes 90% of the time and the trials are independent, the chance that ten in a row all pass is 0.9 to the power of 10, which is about 35%.

How many trials? I don't have a number that suits every team. I'd start with ten per task for anything with side effects and add more for tasks where a failure is expensive. It's a starting point, and you should adjust it once you see how much your results move between batches.

Track which named check fails most. Duplicates point at a retry that replays a write. Boundary failures point at a tool with too much reach. Missing rows point at an instruction that doesn't say clearly what "done" is.

And don't average away a flaky task. A reminder agent that passes nine times in ten sends a duplicate, or nothing, on the tenth. In billing that's too often. Either the harness makes that failure impossible, or the task stays behind a human checkpoint.

The pass rate isn't only a quality number. It's the bottom half of a cost number.

Cost per successful task is cost per run divided by success rate. Cost per run covers every model call, retry and tool call in the task, plus review minutes where a person checks the output. The success rate is the pass rate from these evals. Without it, you can't compare two models or two prompts in any way that holds up.

A smaller model often lowers the cost of a run and the pass rate together. Whether it's cheaper overall depends on the division. And when the pass rate drops after a prompt change, each result has become more expensive even though the provider bill looks flat. That's the moment to roll the prompt back.

I wrote up the cost side, with a worked example, in cost per successful task.

A failed eval tells you where to look, provided the checks have names.

Duplicates. Fix the write path. A second attempt has to leave one reminder and one queued email, and that's a job for a constraint or an idempotency key. Prompts don't prevent duplicates.

Boundary. The tools reach too far. Narrow the permission scope, split a broad tool into smaller ones with typed arguments, and remove access the task never needs.

Missing rows. The instruction is vague or the agent stops early. State the definition of done in the prompt: which row, linked to which invoice, at which step.

Tone fails while state passes. Fix the template or the wording rules and leave the harness alone.

Then run the trials again from fresh fixtures and check that the rate holds. If you'd like this set up for your own agents, it's part of what I build under cost and safety rails.

Paste this into a coding agent in the repository that holds your agent. It drafts one end-state eval for one task and doesn't touch production code. Fill in the square brackets first.

Read the draft before you run it, and check the protected set in particular. The agent will list the tables it can see, and you know which ones matter. After that, run the eval before every prompt change.

Goal: Draft an end-state eval for one agent task: [task name, for example "payment reminder for one overdue invoice"].

Inputs: The task definition: [what the agent should do]. The tools the agent may call: [list, or the file where they are defined]. The tables, folders and queues it may write to: [list]. The protected set that must stay unchanged: [list, or "everything else"]. Tone rules, only if a person reads the output: [rules].

Constraints: Check the state after the run, not the transcript. Use the real tools against test data. Do not stub the write path. Anything that would reach a real person must go to a test queue that never delivers. Each trial starts from a fresh fixture and cleans up after itself. Do not change production code, prompts or tool definitions. Show me the plan before you write files.

Definition of done: (1) A small fixture seed with obviously fake names and fixed dates. (2) A snapshot step that runs before and after the agent. (3) Named assertions in three groups: what must exist, what must not exist (duplicates), what must be unchanged (the protected set). (4) One extra trial that repeats the final write step and asserts there is still exactly one result. (5) A short tone rubric, only if tone rules were given. (6) A runner that executes N trials from fresh fixtures, with N as a setting, and prints the pass rate for this task and a count of failures per named check.