A golden suite for an agent looks reassuring from the outside: dozens of scripted conversations, each with expectations per turn, all green. The question nobody asks while it is green: green for whom? Because if your agent has memory — and every production agent has memory — the answer might be "green for the specific user your cases happen to share, in the specific order they happen to run."
That was the state of a suite I own on a production system. The cases ran as the same synthetic user. The agent's memory did what memory is supposed to do: it persisted. Case B started with residue from case A — preferences the user had "expressed," facts the agent had "learned," context that had no business existing in case B's world. Some cases passed because of that residue. Some failed because of it. The pass rate was not a measurement of the agent. It was a measurement of the agent entangled with an accident of execution order.
Why this failure mode is specific to agents
Classic test-isolation doctrine says: tests must not share mutable state. Every engineer knows it. But in ordinary software, shared state is an implementation smell you refactor away. In an agent system, the shared state is the product feature. Memory, personalization, cross-conversation continuity — the thing that makes the agent good in production is exactly the thing that poisons the suite. You cannot delete it; you have to contain it.
The containment is what I call hermetic cases: every golden case runs as its own synthetic user, with its own seeded state, in its own world. Nothing enters the case except what its fixture declares. The date is the case's date. The user's history is the case's history. When the case ends, its world ends with it.
Two details of the implementation carried most of the value.
Seeded state is declared, not accumulated. A case that needs a returning user with prior context does not get it by running an earlier conversation first — it declares that state in the fixture. This sounds like extra work, and it is, once. In exchange, the case is self-contained: you can read the fixture and know everything the agent knows. Debugging a failure stopped requiring the archaeology of "what ran before this?"
The judge sees the seed. If a case seeds prior state and the judge scoring it does not know that, the judge grades against the wrong world — it flags the agent for "knowing things the user never said." The seeded state renders into the judge's context, so the standard the case is graded against is the world the case actually constructed. Isolation applies to the whole evaluation path, not just the system under test.
The dividend: parallelism becomes free
Here is the part that turned a correctness fix into a speed fix. A suite whose cases share state cannot run in parallel — order is load-bearing, so the suite is condemned to run serially, and as it grows, it slides toward the fate of all slow suites: run nightly, then weekly, then only when someone remembers.
The moment cases were hermetic, order stopped meaning anything, and parallelism required no cleverness at all. First across sets, then across cases within a set — opt-in, because model-backed tests have real cost and you want the throttle in your hand. The full regression went from the slowest step in the pipeline to something you run on every promotion without thinking about it.
That ordering matters and it generalizes: isolation first, speed second. Parallelizing a non-hermetic suite does not give you a faster suite; it gives you a flaky one, because the hidden couplings that were deterministic under serial order become race conditions under parallel execution. Every hour spent on hermeticity was simultaneously an hour spent on parallelism — I just did not know it yet.
The uncomfortable question this leaves you with
If your agent suite has never been through this, the honest question is not "are my cases isolated?" — it is "what is my pass rate actually measuring?" Run the suite in reverse order. Run it shuffled. If the results move, you do not have an evaluation system yet. You have a very expensive way of measuring your execution order.
The fix is not glamorous. Fixtures get longer. Each case pays the cost of declaring its own world. But a golden suite exists to answer one question — did this change break the agent? — and it can only answer honestly when each case's verdict depends on the case, the agent, and nothing else. Hermetic cases are what "nothing else" costs.