An agent fails in production. On turn seven of a long conversation, it promises a customer a refund the policy doesn’t allow. The trace is right there, so the next step looks obvious: reproduce it, fix it, prove the fix. That’s where most teams get stuck.
Agents are inconsistent. Rerun the same conversation from the first message and the agent words turn two a little differently, makes a different tool call on turn four, and never reaches the moment that broke. The bug is real, but it won’t show up on demand. Teams end up fixing what they think went wrong and shipping without proof.
Rerun from the first message, the agent takes a slightly different path and never reaches the moment that broke.
This post explains how Replay in Plurai Simulation solves that. It takes a failure from a production trace and plays it again from the exact turn where it broke, the same way every time. The failure stops being a story in a trace and becomes something you can run.
Why rerunning from the start misses the failure
Every turn is a small chance to drift. The agent phrases something differently, the user answers differently, a tool gets called in another order. Each drift is harmless on its own, but they compound, so the later the failure, the less likely a rerun from the start ever reaches it.
Even when a rerun does reach turn seven, the world it sees may not match. A booking created on turn three has a different ID; a balance is off by an earlier refund. The failure depended on that exact state. Reproducing it means holding two things fixed: the conversation up to the failing turn, and the world the agent saw at that moment.
Principles for reproducing an agent failure
Reproduce the moment, not the conversation. Replaying from turn one tests whether the agent can find its way back to the bug. Start at the turn that broke.
The world is part of the bug. A transcript isn’t a reproduction. The agent failed against specific records and specific tool responses, so those have to come back too.
A failure you can’t repeat isn’t reproduced. One rerun that fails could be luck, and one that passes proves nothing. A reproduction counts only when it fails consistently.
Test the fix where it failed. A fix is proven only when the changed agent faces the exact same moment and gets it right.
How Replay works
Rebuilding the world
Replay starts from a production trace, pulled from your observability tool. The trace already holds what matters: every message, every tool call and every tool result. The agent runs inside a digital twin where every tool it touches is mocked, so each tool can answer exactly as it did in production, backed by the records those answers came from. Turn by turn, the world is rebuilt to the state the agent saw right before it failed.
Every value the agent saw in production comes back: the booking, the fare, the rule it broke. Then it writes turn seven again.
Starting at the failing turn
The agent doesn’t regenerate turns one through six. It receives them as history, exactly as they happened, with the world frozen at that point. Then it produces turn seven, the one that broke. Nothing before the failure can drift, because nothing before the failure is rerun.
History and world state are fixed from the trace. Only the failing turn runs, three times.
Making it deterministic
The agent is still inconsistent; that doesn’t go away. So the frozen moment runs three times. Three failures out of three confirm the bug and give the fix a clear target. One or two tell you it’s intermittent, and an intermittent refund violation is still a refund violation.
Replaying against a changed agent
Replay stays compatible when the agent changes. Edit the prompt, swap the model or change a tool, and the new version faces the same frozen moment: same history, same world, new agent. If it passes where the old one failed, the fix works on the exact case that broke in production.
What a replay found
Here’s a real replay. A design partner runs a billing-support agent for their SaaS product. It verifies the customer, looks up their account and invoices, and can change plans, cancel subscriptions and issue refunds, but only up to $200; anything larger goes to a human. Here are two failures from its traces. Each became a replay: cut just before the reply that went wrong, with the history restored word for word and the next customer message sent live.
The $550 refund came back every time. A customer on an annual plan wanted to cancel and asked for money back on the unused months. The agent worked out the right amount, $550 for 11 months, and offered to process it itself. That’s the failure: it can refund at most $200, so it should have stated what the policy grants and escalated the rest. All six replay trials made the same offer. The production trace ended there; the replay didn’t. The simulated customer said yes, and in two trials the agent actually called the refund tool for $550 before the limit stopped it. Along the way, the replay surfaced problems the trace never reached: cancelling without a clear yes, skipping the retention offer, and telling the customer the cancellation was immediate.
What it was worth. Before the replay, this was one odd line in one trace. After it, it was a confirmed bug that fails six times out of six, and a more serious one than the trace showed: pushed by the customer, the agent will try to move money past its limit. That turns a backlog item into a fix for today, with a precise target: the agent has to know its refund authority and escalate above it. The replayed moment then stays in the suite as a regression, so every future prompt or model change is checked against the exact conversation that broke.
The failed verification came back in part. A customer asked for copies of their invoices, but the workspace ID they gave didn’t match their email. The rule: share nothing, explain the mismatch, and after a second failed attempt, stop and open a ticket. The agent got the first part right and never leaked an invoice. But in every trial, it kept asking for more details after the second miss.
What it was worth. “Verification failed” in a trace says very little; it could just be the customer’s typo. The replay separated the two: the wrong workspace ID was the customer’s, but the defect was the agent’s next move. That narrows the fix to a single rule, stop after the second miss and open a ticket, and the regression scenario checks that rule on every run.
From replay to regression
A reproduced failure shouldn’t be fixed once and forgotten. Each replayed moment can be saved into the simulation suite as a regression scenario, with its history and world state intact. From then on it runs with everything else, so a later change that quietly brings the bug back is caught at the exact turn it first appeared.
That closes a loop that’s usually open. Simulation finds failures before production; Replay brings production failures back into simulation. Both end in the same stable suite.
Make failures runnable
Agents will stay inconsistent. That’s what makes them flexible, and it’s what makes their failures so hard to pin down. The answer isn’t to make the agent deterministic. It’s to make everything around it deterministic, so the only thing that changes between runs is the thing you’re trying to fix.
Hold the world still, and the bug has nowhere to hide.
Replay is available in Plurai Simulation today. Connect your traces, pick a failure, and run it again.
