Agents are now easy to build and change every week. The bottleneck is verification: knowing the agent works, and that fixing one behavior didn't break four others. You can only verify what you test.
And the test space explodes. Take a travel assistant with 15 flows, jobs it does end to end, like changing a booking, paying a fare difference or applying a voucher. "Move my flight to Thursday" can touch three of them in one conversation, and any three of 15 can meet in 455 ways. Agents with no conversation work the same way: a claims workflow that reads a document, checks coverage and routes an exception crosses three flows too.
Each combination then varies with the user, the data, the tools and the rules, and even a reworded request is a new test. One agent ends up with over two million scenarios.
No team can test all of that. The largest public benchmark for conversational agents covers 115 scenarios per domain.
This post shares the approach behind Plurai Simulation: model the agent as a flow graph, weight it with probabilities, and sample along the dimensions that change outcomes. The approach grew out of IntellAgent, our research on evaluating conversational agents. The goal isn't to test more. It's to test what matters most.
The question behind every simulation
A simulation engine doesn’t test one agent. It tests whatever a team points it at, an airline assistant today and an IT helpdesk agent tomorrow, so there’s no fixed test set to curate once. Every scenario is a full multi-turn run with a simulated user, a world state and a judge, and teams rerun after every change. Choosing scenarios is a daily budget decision.
The first thing teams ask us is “how many scenarios do I need?” It’s the wrong question. A thousand happy-path scenarios tell you less than fifty that each hit a seam.
This isn’t new in simulation. Autonomous driving faced it at a much larger scale: the space of road situations is effectively infinite, and simulated miles don’t help if they’re the wrong miles. The field moved from counting miles to scenario-based testing, choosing scenarios by how often they occur and how dangerous they are. Agents are on the same path, and much of what follows borrows from that playbook.
A million highway miles won’t find the cyclist at the intersection. A thousand happy-path chats won’t find the refund that skips identity verification.
Volume also blurs what it does find. Most suites end in one pass rate, which says something failed but not which rule, which combination, or how hard the scenario was. So we asked: what if test generation started from a map of what the agent does, and how those things combine?
Principles for deciding what to test
Test the joints, not the parts. Most suites test one intent at a time. Agents rarely fail at a single flow in isolation; they fail when two or three flows share a conversation and their rules collide. So the unit of testing is a combination of flows.
Own your test space; don’t borrow it. Generic red-team catalogues and persona libraries test what agents in general get wrong. Your failures are specific to your rules, tools and knowledge bases, so the test space is compiled from them: every rule becomes a constraint, every tool a way to fail, every knowledge-base article a fact the agent can get wrong.
Optimize for risk covered, not scenarios run. Scenario count is the metric everyone reports and the wrong one to optimize. Each scenario’s chance of being drawn is weighted by how likely its flows are to meet and how hard they are, so the budget concentrates where the risk is.
Find the breaking point, not the pass rate. A pass rate averages easy and hard scenarios into one number that hides where the agent fails. We control complexity directly and run across the full range, so you see the level at which the agent starts to break, and whether a change moved it.
Check what got done, not what the agent said. An agent can say “your refund is on its way” without ever calling the refund tool. Every scenario runs against a seeded world state and carries its own ground truth from your rules and knowledge bases, so we check the end state: was the booking actually changed, the refund actually issued, the answer drawn from the right source.
One scenario, annotated. Each number marks where a principle shows up: the flows it combines, the sources it was compiled from, how often it happens, how hard it pushes, and the end state it’s checked against.
How it works
Under the hood, “what should we test?” breaks into three questions the engine can answer from the agent alone: which flows meet, how hard to push them, and how much budget each combination deserves. The sections below take them in turn, then show how a sampled scenario actually runs.
The flow graph
The first step is mapping the agent’s behavior: its instructions, tool definitions, knowledge base and data together describe the jobs it’s meant to do and the rules around them. We turn that description into a flow graph.
Each node is a flow: the smallest unit of work with its own definition of success. “Issue a refund” is a node; a single API call is too small, and a whole conversation is too big. Every node carries what triggers it, the tools it calls, the rules that constrain it (“refunds over $500 need supervisor approval”), the knowledge-base content it depends on, and the end state that counts as done.
Each edge links two flows that can meet in one interaction. Edges come from what flows share: a precondition (a refund needs identity verified first), an output one flow hands to another (a booking change produces a fare difference), or a tool or record they both touch. Each edge also carries a weight for how often the two flows meet, which the next sections put to work.
Part of a flow graph for an example travel assistant. Edge thickness reflects how often two flows appear together; the highlighted node is where a failure costs the most.
Since failures happen where flows meet, the edges are where we look first.
Dimensions
A path says which flows a scenario exercises, not how hard they’re pushed. So every scenario is also a point along five dimensions:
These compound: a complex scenario on messy data with a flaky tool is harder than any of them alone. Every scenario runs against a seeded digital twin of the agent’s environment.
Choosing paths
Weights turn the graph from an enumeration into a decision about where to look.
Edge weights estimate how likely two flows are to co-occur. Node weights estimate how hard and costly a flow is; issuing money weighs more than looking up a flight.
Paths are chosen by walking the graph. Each walk draws a target complexity, starts at a flow, and follows edges in proportion to their weights until it reaches that complexity. Heavy edges get walked often, high-impact nodes pull walks toward them, and unlikely, low-impact combinations rarely get drawn.
Why weighted, and not simpler? In our research, uniform walks strung together flows that never meet in real life, and always taking the heaviest edge collapsed every walk into the same cluster. Weighted walks sit between: realistic, and still diverse.
Three rules shape the suite beyond the weights. Complexity is spread on purpose: every suite holds short and long paths in a set mix, so results read as a curve over difficulty, not a single number. Near-duplicates are collapsed: two paths through the same flows in the same way count once. Failures are kept: a path that failed in an earlier run, or in production, stays in the suite as a regression scenario.
From path to run
A sampled scenario then runs in four steps:
- Seed the world. Generate the booking, passenger and fare rules it needs, and load them into the mocked tools.
- Run. A simulated user, or an upstream trigger, drives the agent; the digital twin answers its tool calls.
- Judge. Score the run against the specific rules the scenario was built from.
- Report by node. Roll failures up to the graph, by flow, combination and complexity.
It all runs from one command in the plugin, in Claude Code, Cursor, Codex or Copilot.
One scenario, drilled into: the agent states a date its own lookup contradicts, and the judge flags the exact line.
Found the issue. Now what?
Most testing stops at a red mark. Here, every scenario starts from its answer: the rules, sources and end state it was built to check. So a failure isn’t just a bad conversation. It points to a specific rule on a specific node, and from there to the part of the agent that governs it.
That’s why each gap comes with four things:
- Diagnosis: why the agent did what it did, in plain language.
- Location: where that behavior lives, down to the file and line in the prompt or tool.
- Proposed fix: the concrete change, such as a paragraph to add to the system prompt.
- Regression: the failing scenario, kept so the next run confirms the fix held and didn’t break the flows around it.
From failure to fix: the gap is traced to its route in the flow graph, diagnosed down to the line in the system prompt, and recorded as a regression scenario for the next run.
Fixing is where agents regress. Change the prompt to close one gap and three other flows can shift with it. So validation has to be stable: the same scenarios, against the same seeded world, on every run. A green rerun then means the fix held and nothing around it broke, not that the dice rolled differently.
On design-partner agents, the proposed fixes cleared most of the failures they targeted, with suites of just eight scenarios each.
What we keep seeing
A few patterns keep showing up across agents:
Every model breaks; the question is where. We keep seeing models that differ on simple scenarios converge at high complexity, where nearly all fail. The breaking point is the number worth comparing, not the average.
“Which model?” has no general answer. We rarely see one model lead across every flow type and complexity level. The right model is a property of your agent, not of a leaderboard.
The worst failures read like the best conversations. In the run shown above, the agent is confident, polite and wrong: it states a flight date its own lookup contradicted. Graded end to end, that conversation looks like a success. Checked claim by claim against what the tools returned and what the rules require, it fails on one line. These are the failures manual review routinely misses: a refund issued without consent, a tool error smoothed over instead of recovered from.
Synthetic can agree with humans. In the IntellAgent paper, success rates on 1,000 graph-sampled scenarios per domain closely tracked τ-bench’s hand-curated 50 (airline) and 115 (retail), across six models: r = 0.98 and 0.92.
Review the map, not the transcripts. Nobody reads a thousand conversations. One review of the graph’s flows, rules and weights fixes every scenario built from it.
Where this fits, and where it doesn’t
The graph is only as good as what it’s built from: a rule the agent’s definition never mentions won’t be tested. Weighted sampling also gives rare, low-impact combinations little budget by design, so some failures will surface in production first. That’s why simulation pairs with trace validation: pull traces from LangSmith, Langfuse or Arize and replay failing turns against the digital twin. And “impact” is a business call. The graph proposes weights; your team owns them.
Test what matters most
Agents will keep getting faster to build. Every new tool, rule and model swap widens the space of what can go wrong, and no team will ever test all of it.
The teams that ship with confidence won’t be the ones that test the most. They’ll be the ones who know where to look.
~~~
Plurai Simulation is available today. Point it at your agent, run one command, and see where it breaks.
