Evaluating agents beyond the final state.

When I evaluate an agent, I treat its last message as an unverified claim. A payment agent can say that a refund is complete, and the account can even show the expected balance. The event log might still show that the agent moved money without approval and then repaired the balance with a second transfer. A final-answer check sees a good response. A final-state check sees a good database. Both accept a run I would reject.

Agent evaluation needs a wider record: the state the agent changed, the events it caused, and the variation across repeated attempts. I also want the checks to permit different valid solutions. Comparing every tool call with one reference transcript mistakes a particular solution for the task itself.

I use a split between goal predicates, trajectory constraints, and protected state. This post works through that split, along with session design, reliability metrics, and the limited role I give to model judges. The examples come from web, software, computer-use, and tool-use benchmarks, but the same harness applies to a small internal evaluation set.

Evaluate environment state, not action transcripts

An action transcript records what the agent sent to its tools. The environment state lives elsewhere: in database rows, files, calendar entries, repository contents, or an application’s internal configuration. Evaluation should inspect that state directly whenever the interface allows it. A screenshot or a confident closing message is weak evidence about those objects.

WebArena made this shift for web tasks by checking functional correctness in self-hosted applications. Its evaluators locate content or inspect underlying state, so two agents can reach the same goal through different routes (Zhou et al., 2023). SWE-bench applies the same idea to repositories. A patch must make designated fail-to-pass tests succeed, while other tests check that existing behavior remains intact. The original tasks have a median of 51 additional tests beyond the fail-to-pass checks (Jimenez et al., 2023). Those regression tests are a form of protected-state evaluation.

AppWorld gives the cleanest formulation for an API environment. It computes the database diff between the initial and final states. Every expected change must be present, and changes outside the expected and explicitly allowed sets cause test assertions to fail. Its unit tests avoid requiring one reference action sequence while still catching unexpected writes (Trivedi et al., 2024).

I would copy that structure for most tool agents. Start from a snapshot, execute the run, compute a semantic diff, and write assertions about the diff. IDs and timestamps may need normalization. Everything else should be expressed in domain terms, such as one refund for the requested order, rather than in terms of the agent’s chosen API calls.

Separate goals, ordering, and protected state

State checks answer whether the run ended well. Event checks answer what happened along the way. I keep three kinds of predicates in the evaluation record:

• Goal predicates describe required outcomes, such as a refund issued to the original payment method.
• Trajectory predicates describe required events or ordering, such as explicit approval before the refund.
• Protection predicates cover unrelated objects and temporary conditions, such as no change to another order and no balance below the reserve at any point.

The temporary conditions matter because a final snapshot can erase them. CORE models valid tool-use paths with deterministic finite automata and scores path correctness, harmful calls, and efficiency. Its transaction example is a transfer followed by a compensating transfer. The balance recovers, but the two operations are not atomic, so an interruption can leave the system in the wrong state (Michelakis et al., 2025).

A schematic evaluator can stay small:

def evaluate(run):
                        checks = {
                            "goal": run.state["refund"]["status"] == "issued",
                            "approval_first": run.before("approval", "refund"),
                            "reserve_held": run.minimum_balance() >= RESERVE,
                            "only_allowed_writes": run.state_diff() <= ALLOWED_CHANGES,
                        }
                        return checks

I return the individual checks even when strict success requires all of them. That separates an agent that missed the goal from one that completed the goal through a disallowed route. τ-bench illustrates the distinction in the other direction: its authors note that a correct final database and required user-facing output can still miss a policy violation, such as issuing a return without confirmation (Yao et al., 2024).

Trajectory predicates should remain sparse and semantic. Requiring an approval event before a write is different from requiring one exact chain of reads and writes. The first encodes a rule. The second often encodes the benchmark author’s preferred solution.

Design sessions with memory and consequences

A task that supplies every fact in the first message and ends after one write tests a narrow part of agent behavior. Longer sessions let earlier actions constrain later choices. They also let me check whether an edit persists after the agent turns its attention elsewhere.

Useful session variations include:

• Start in the middle of a workflow, with existing files, messages, and partially completed work.
• Introduce a new instruction after an earlier edit, then verify both the new request and the surviving edit.
• Return a tool error or incomplete information and check whether the agent recovers without inventing a result.
• Include an impossible request and require abstention with no state changes.

OSWorld provides a useful implementation pattern. Each task has an initial-state configuration, an executable environment, and a custom evaluator that retrieves files or application data after the run. Virtual-machine snapshots make reset possible even when the agent controls a full desktop (Xie et al., 2024). I would also freeze the clock and seed any user simulator. A task that depends on today’s date or a changing website will otherwise produce moving labels.

Simulators help construct rare situations. ToolEmu uses a language model to emulate tools and to generate adversarial test conditions around underspecified instructions. The authors validate both the emulations and the identified risks with human review (Ruan et al., 2023). For a regression suite, I would replay important scenarios against deterministic tool implementations. A generated emulator can forget a deleted file or return an impossible transition, and that error can become the agent’s score.

Intermediate labels help diagnose stalled runs. OSWorld-Pro adds more than 2,800 subgoals to over 300 computer-use tasks, grounded in more than 67,000 human annotations (Wang et al., 2026). I would use subgoal scores for analysis and keep strict task success visible alongside them. Partial progress can otherwise make an unfinished run look close to complete when a late dependency blocks the goal.

Measure reliability across repeated runs

One run measures one sampled trajectory. Model sampling, a simulated user, network failures, and small interface changes can all alter the next run. Repeating tasks is part of the evaluation rather than a way to decorate the score with a confidence interval.

The two common k-run metrics answer different questions. pass@k asks whether at least one attempt succeeds, which fits candidate generation followed by selection. pass^k asks whether all k attempts succeed, which fits an agent that must handle the same request every time. Under independent attempts with success probability p, that probability is p^k. At p = 0.7, eight consecutive successes have probability 0.7^8 = 0.0576.

τ-bench introduced pass^k for tool-agent-user interaction and estimated it across repeated trials of each task. In its original retail experiments, GPT-4o averaged about 61% single-run success but fell below 25% on pass^8 (Yao et al., 2024). That gap describes reliability more clearly than either the best run or the average run alone.

I report single-run strict success, an all-runs metric, and the distribution of per-task success rates. I also keep goal success separate from success with every constraint satisfied. A model can improve the first while making more unauthorized writes on the way to the goal.

Tool calls, elapsed time, tokens, and cost belong in the same run record, with medians and upper quantiles. Long traces can mean exploration, repeated retrieval, or a loop, so call count needs the event log for interpretation. Every compared system should receive the same step and time budgets. Changing the cap changes the task, especially when a run needs recovery steps after an error.

Keep model judges on narrow semantic checks

Anything that can be checked from a database or event log should stay in code. I reserve model judges for semantic questions that resist a fixed predicate: whether a user-facing explanation covers the required points, whether a summary preserves the source, or whether a requested piece of text has the specified properties.

The judge receives a narrow question and the relevant evidence. It should not receive a whole trajectory with an instruction to decide whether the run looks good. AgentHarm follows this division by using predefined conditions for most grading and semantic judges for isolated parts of an output (Andriushchenko et al., 2024). Fixed rules can then combine those observations with the programmatic checks.

A judge needs its own evaluation. Work on chat grading documented position bias, verbosity bias, and possible preference for a judge’s own generations (Zheng et al., 2023). Agreement on open-ended chat answers does not establish agreement on agent trajectories, where the judge must track tool results and ordering.

I build a small labeled set containing clear passes, clear violations, and borderline traces. Pairwise judgments run with the candidates in both orders. A sample of single-answer judgments gets human review, with disagreements used to revise the rubric. The model version, prompt, and scoring code stay pinned for a comparison.

Agent-produced text in the judge input is untrusted content. A run can contain instructions aimed at the judge just as a web page can contain instructions aimed at the agent. Structured evidence fields and narrow output schemas reduce ambiguity, though they do not remove the need to inspect odd judgments.

A minimal harness for your own tasks

A compact harness can store almost everything needed for this style of evaluation. I would begin with three records:

• Task specification: initial snapshot, seed, instruction schedule, expected changes, required events, protected conditions, and run budget.
• Execution record: model and harness versions, tool arguments, tool results, timestamps, errors, state snapshots, and the final response.
• Evaluation record: each predicate result, the state diff, resource use, and any judge observations with their evidence.

Before comparing agents, I replay a reference solution and require every goal check to pass. I also mutate that run into targeted violations: remove the approval, change an unrelated row, exceed the reserve for one step, or repeat a write. Each mutation should trip the intended predicate. A check that has never failed on a constructed bad run is unfinished.

Then I read a sample of traces by hand. I look for false rejections of valid alternative routes and for accepted runs that violate the task in a way the predicates missed. New checks go back into the same suite, so the evaluation improves without changing the meaning of the task instructions.

For a first internal benchmark, I would choose fewer tasks with strong reset logic and thorough assertions over a large collection scored by a general prompt. The main table can remain simple: strict success, goal-only success, constraint violations, and repeated-run reliability. The stored records provide the detail behind those numbers.

The final answer still has a place in this harness, especially when the user needs information. It is one artifact of the run. The state diff and event history show what the agent did to produce it.

Sources and further reading

Executable agent benchmarks

• Zhou et al. (2023). WebArena: A Realistic Web Environment for Building Autonomous Agents. ICLR 2024.
• Jimenez et al. (2023). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. ICLR 2024.
• Trivedi et al. (2024). AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents.
• Xie et al. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments.
• Yao et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.

Trajectory and safety evaluation

• Michelakis et al. (2025). CORE: Full-Path Evaluation of LLM Agents Beyond Final State.
• Wang et al. (2026). OSWorld-Pro: Process-based Evaluation for Computer Use Agents.
• Ruan et al. (2023). Identifying the Risks of LM Agents with an LM-Emulated Sandbox. ICLR 2024.

Judges and rubrics

• Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023.
• Andriushchenko et al. (2024). AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents. The grading rubrics are a useful example of reserving semantic judges for narrow checks.

All posts