How faithful is a test rebuilt from traces?

When an agent does something wrong in production, the trace is usually the best record of what happened: what the customer asked for, which tools the agent called, and what each tool returned. That makes it tempting to turn the trace into a test. Keep the customer and the tool answers, change the agent, and play the conversation again.

The hard part is knowing whether the replay can be trusted. If the rebuilt test fails where production failed, the rebuild might be faithful, or it might be broken in a way that happens to fail. If the fixed agent now passes, there is no way to tell whether it would pass for real. Without an outside reference, a test rebuilt from traces is grading itself.

τ²-bench gives us that reference. Sierra published full conversations between airline support agents and a simulated customer, the environment those conversations ran in is open source, and every task has a grader that inspects the final database. We rebuilt tests from the published conversations using only what their traces recorded, ran them again, and asked the real environment whether the re-runs came out the way production did.

What we found:

• Where no tool answer had to be simulated, the rebuilt test got production's verdict in 47 of 58 conversations (81%). Two production runs of the same task agree with each other 79% of the time. We can't tell these apart.
• The misses come from answers the trace didn't have. When the agent made a call no recording covered, a model wrote the answer, and most of those answers were wrong. Lookups of a specific reservation were wrong 70 times out of 71.
• The rebuilt tests did not predict a model switch. After swapping gpt-4.1 for gpt-4.1-mini, the per-task predictions were barely better than assuming nothing would change.
• What the rebuilt customer remembers matters as much as the tools. Two reasonable rules for it gave 63 and 75 matches out of 100.

The setup

The conversations

We used the airline runs published with τ²-bench v0.1.3: 50 tasks, four trials each, with gpt-4.1 as the agent and gpt-4.1 at temperature 0 playing the customer. That is 200 conversations, 112 of which pass. In the rest of this post they are "production". The same release has gpt-4.1-mini on the same tasks (101 of 200 pass), which we use later as the real outcome of a model switch.

Before relying on the grader, we re-scored all 400 published conversations with τ²-bench v0.1.3. Every one matched its stored reward.

What the rebuild could see

A team testing its own agent has two things: its traces and its code. The rebuild got the same two. It could read each conversation in full, meaning the customer's messages, the agent's messages and every tool call with its arguments and result, and it could read the agent's code: the system prompt, the airline policy and the tool definitions. It never saw a task's hidden scenario, the grader's criteria or the database.

We added one restriction. The four trials of a task are the same customer arriving at the same moment four times, which never happens in production. So a conversation's test could borrow examples from other tasks' conversations, but never from the other three trials of its own task. This restriction turns out to matter, and we come back to it below.

From a conversation to a test

We wrote each conversation out as an OpenTelemetry trace, one session per conversation, which is roughly what a team would export from Langfuse, LangSmith or an OTel collector. ancha import --offline then made one test per conversation on the local machine. Each test has three parts.

The customer. A model (gpt-4.1-mini) reads the customer's messages, with the agent's messages as context, and writes down a goal and a list of facts. During the test a small model (Gemma 4 26B) plays the customer from that description and gives a fact only when asked for it. This is what the importer wrote down for the customer in task 5, trial 0 (an excerpt, with the long goal line wrapped):

people:
  - ref: customer
    goal: You want to address the delay of your flight HAT045 from Phoenix
      to Seattle, which caused you to miss an important meeting. As a Gold
      member, you expect the maximum possible compensation, preferably more
      than a travel certificate and ideally a refund to your original payment
      method. You want to resolve this without being transferred to another
      agent, but if no better compensation is possible, you will accept a
      travel certificate applied to your account.
    knows:
      - id: user_id
        value: mei_brown_7075
        about: your user ID
      - id: flight_number
        value: HAT045
        about: the flight number that was delayed
      - id: flight_route
        value: Phoenix to Seattle
        about: the route of the delayed flight
      - id: membership_status
        value: Gold member
        about: your loyalty status with the airline

The tools. Each call the agent makes is answered from the best source available, and every answer records where it came from. A call this conversation made in production is replayed. The same call made in another conversation is replayed from there. A tool with no side effects, which here is only calculate, is simply run. Anything else is answered by a model that has seen how the tool answered in other conversations. We call these simulated answers.

Writes. Calls that change something, like booking, cancelling or updating a reservation, never execute. The agent gets a plausible confirmation, and the call is judged afterwards by what it would have done to the real database.

Running and grading

The agent is τ²-bench's own LLMAgent at temperature 0, unchanged, with its tool calls routed to the test instead of τ²-bench's environment. To measure reproduction we re-ran trials 0 and 1 of every task with gpt-4.1, the production agent, which gives 100 conversations. For the model switch we re-ran all 200 with gpt-4.1-mini.

Grading used τ²-bench's evaluator, which checks the final database and whether the agent told the customer what it had to. No model is involved in grading. One step needed care. The evaluator replays the agent's calls against the real database and refuses any conversation in which a tool returned something the real database wouldn't have. That mismatch is exactly what we want to measure, so we first swapped every tool answer for the real one, counting how many the test had right, and then scored what the agent's own actions did to the real database.

The model calls for all 300 re-runs cost about $5.

Replaying production

Re-run with the same agent, 75 of the 100 conversations got the verdict they got in production. Splitting them by where their tool answers came from shows where the gap is.

Two production runs of the same task79%238 of 300 pairs

Rebuilt, no simulated answers81%47 of 58

Rebuilt, all conversations75%75 of 100

Rebuilt, at least one simulated answer67%28 of 42

In the 58 conversations where no answer was simulated, 47 matched (81%). In the 42 that needed at least one simulated answer, 28 did (67%). The reference point is production itself. Take two of the four published runs of the same task and they agree on pass or fail 79% of the time. The agent and the customer both run at temperature 0, and their conversations still drift apart enough to change the outcome about one time in five.

With 58 conversations the 95% interval around 81% runs from about 69% to 89%, so the fair reading is that a rebuilt test without simulated answers agrees with production about as often as production agrees with itself. We wouldn't expect better. A rebuilt test that matched one production run more often than a second production run does would be fitting that run's noise.

The misses are not symmetric. Production's failures failed again in 36 of 44 conversations (82%), while its passes passed again in only 39 of 56 (70%). The rebuilt tests were harsher than production: 47 passes against production's 56. For a regression test that is the safer direction to be wrong in, but it still means a failure in a rebuilt test is worth reading before anyone acts on it.

Where the misses come from

Every answer in a re-run records its source, and τ²-bench lets us check each one against what the real database would have returned.

Recorded answers were wrong 8 times out of 1,238. Simulated answers were wrong most of the time, and which tool was being simulated explains most of the difference.

A model can write a correct answer to a transfer request, because that answer carries no facts. It only says the transfer happened. It can't know what is in a particular reservation, or which flights actually leave on a given day, so it writes something with the right shape and the wrong contents. Write results go wrong for the same reason. The answer to a cancellation describes the reservation that was cancelled, and the model has never seen that reservation.

So where do the unrecorded calls come from? About a third of the wrong answers (15 of 49 for gpt-4.1, 61 of 185 for gpt-4.1-mini) were the exact call that one of the task's other three production runs had made. Those are the runs we withheld. A team that imports all of its traces would have had these answers replayed. The other two thirds are reads that no production conversation made: a reservation looked up for the first time, or a search on a date nobody had searched.

This is why we report, next to every rebuilt verdict, how much of it rests on simulated answers. Ancha prints it for each test, as in 1/3 verdicts rest on simulated answers. The tables above are a good reason to discount any verdict where that count isn't zero.

Writing down the customer

The tools are half of the rebuild. The other half is the customer, and the two ways we got the customer wrong taught us more than the tool numbers did.

Our first rule wrote the goal and facts from the customer's own messages, using the agent's messages only as context. That sounds safe, but customers repeat what they are offered. In task 5 the customer is a Gold member whose flight was delayed. In production, gpt-4.1 offered a $200 travel certificate and the customer accepted it in their own words, so the $200 certificate ended up in the rebuilt customer's goal. Re-run against gpt-4.1-mini, the rebuilt customer asked for the $200 certificate by name, and mini issued it. Under the airline policy a delayed flight earns a certificate only if the customer changes or cancels the reservation, so the test failed. Real gpt-4.1-mini passes task 5 in all four of its published runs: it asks whether the customer wants to change or cancel, the customer says no, and nothing is owed.

So we changed the rule. Anything the agent said before the customer did was dropped from what the customer knows. That fixed task 5, where the re-run mini now passes four out of four like the real one, and overall reproduction fell to 63 of 100.

The new rule dropped too much. In task 8 the agent had read the customer's details out of their profile before the customer mentioned them: their companion's date of birth, a travel certificate on the account, and the flight they went on to pick. Those values appear first in the agent's messages, so the rule threw them away, and the rebuilt customer no longer knew things any real customer would know about their own trip. Compared with the first rule, 18 conversations stopped matching production, all of them with no simulated answers, and 16 of the 18 were missing facts in this way.

The rule we ended with separates the two cases. A value the agent read out of a tool's answer stays with the customer, because it comes from their own records. A value no tool returned, such as an offer the agent worked out, is dropped. Reproduction went back to 75 of 100, and task 5 mostly stayed fixed: the re-run mini passes it three times out of four.

Predicting a model switch

The question we started from was what happens when you change the agent. So we swapped gpt-4.1 for gpt-4.1-mini, re-ran all 200 conversations, and compared the results with mini's real published runs.

The rebuilt tests did not predict the switch. Per task, the re-run pass rate was 0.270 away from mini's real pass rate on average. Assuming nothing changes, that is, using gpt-4.1's real pass rate as the guess for mini, is 0.285 away. For scale, chance alone would put two four-trial runs of the same agent 0.100 apart per task (a binomial estimate from mini's real pass rates), and that is the floor for any prediction made from four runs per task. On the 16 tasks where the switch moved the real pass rate by at least two trials out of four, the rebuilt tests got the direction right on 7.

Two things work against prediction. A different agent takes a different path, and a different path means more calls that production never made: mini's re-runs needed 1.4 simulated answers per conversation, against 0.8 for gpt-4.1, and simulated answers are the ones that tend to be wrong. The customers were also written down from gpt-4.1's conversations. Even with the rule above, they bring some of gpt-4.1's path into mini's conversations.

There is also less to detect than it might seem. On the same 100 conversations, the real switch moves the pass count from 56 to 51, about one standard error. Per task, most of what changes between the two models is noise. We don't think tests rebuilt from traces should be used on their own to choose between models. They answer a narrower question well: whether a new version of the agent still does what production did, in the conversations production actually had.

A note on grading

We also ran Ancha's default judge, a model that reads the transcript and decides whether the customer's request was resolved, and compared it with τ²-bench's database grader. They agreed on 63 of the 100 gpt-4.1 re-runs and 114 of the 200 gpt-4.1-mini ones. A conversation can read as resolved while the database shows the change was never made, or made wrong, and an awkward conversation can leave the database exactly right. When the outcome lives in a system of record, check the record.

What we take from this

• Replaying what production recorded is faithful, to about the level of production's own variation.
• Import every trace you have. A third of the wrong answers here were sitting in traces we had withheld.
• Count the simulated answers behind each verdict, and give little weight to verdicts that depend on them.
• Be careful about what the rebuilt customer remembers. Keep what came from their own records, and drop what the old agent offered.
• Don't use rebuilt tests alone to predict a model switch.

Limitations

This is one benchmark and one domain. τ²-bench's production is itself simulated: the customers in the published runs are gpt-4.1 following a written scenario, which makes them more consistent than real people. Real traces will differ in both directions, with more variety in what customers say and probably fewer exact repeats of the same tool call.

The samples are small. Reproduction rests on 100 conversations, the comparison with production's own variation on 58, and the switch result on the 16 tasks that move.

The rebuild had the agent's code, as a team testing its own agent would. It did not have τ²-bench's database, which is the point of the exercise: everything it knew about the data came from the traces.

Reproducing this

• Conversations: the τ²-bench v0.1.3 airline submissions for gpt-4.1 and gpt-4.1-mini, four trials per task, from the public sierra-tau-bench-public bucket.
• Environment and grader: τ²-bench v0.1.3, evaluate_simulation with all checks, after replacing each tool answer with the real one.
• Ancha CLI 0.4.0: ancha import --offline with calculate marked as a pure tool, one test per conversation, then ancha run with one trial per test.
• Models, all through OpenRouter: the agent is gpt-4.1 or gpt-4.1-mini at temperature 0, gpt-4.1-mini writes down each customer, and Gemma 4 26B plays them.
• Cost: about $5 of model calls for the 300 re-runs.

The importer is documented in ancha import.