Putting Harness² on call.
Google Research's harness optimizer rewrote an incident agent's instructions and took a failing task from 0 of 3 to 2 of 3 passes. Every probe ran in a fresh world with state, and every run was checked against what the agent actually did.
Much of what makes an agent reliable lives outside the model: the system prompt, the skills it can load, the rules for when it is done. Engineers tune this layer by hand, one transcript at a time. Harness², released by Google Research in October, automates the tuning. It keeps the model fixed and edits the harness around it.
An optimizer is only as good as the evidence it learns from. We wanted to see what Harness² does on a task with consequences, where a fluent final message and a finished job are different things. So we pointed it at an on-call scenario in Ancha. Every probe started a fresh world with working replicas of Slack, GitHub, Datadog and PagerDuty, and every run was graded on what the agent did to that world.
With the paper's default editor, Claude Opus 4.8, Harness² found a one-paragraph change that fixed the failure in two of three fresh incidents. With Claude Sonnet 4.6 as the editor, its edits looked reasonable and did not move the score, and its composer described failing runs as successful. Checks on API calls and world state are what told the two apart.
How Harness² works
Harness² treats eight parts of a harness as editable: the system prompt, guardrails, memory, tool guidance, subagents, skills, scripts and plugins. Its core step is propose, probe, compose. The starting harness runs on a task. An editor reads the task, the trajectory and the deliverable, then proposes two changes in separate sessions, one general and one specific to the domain. Each candidate runs on the same task. A composer reads both candidates and how they behaved, writes a successor harness and runs it.
The starting harness works the incident in a fresh world.
Reads the task, trajectory and deliverable. Writes a general edit and a domain edit in separate sessions.
Each edit works the same incident, each in its own fresh world.
Reads both edits and how they behaved, writes a successor harness and runs it.
Two design choices stand out. The loop has no verifier. Candidates are built from how executions differ, which lets the method target tasks that come without labels. Nothing is thrown away either. Candidate files, traces, notes and composition decisions stay on disk, so an edit can be read, rerun and reverted like any other change to code.
A single step is more compute than the name suggests. The paper (arXiv:2610.14224) reports single-step gains of up to 11.5 points on WorkBuddy-Bench, 4.5 on JobBench and 11.8 percentage points of per-criterion partial pass rate on Harvey's Legal Agent Benchmark, with Gemini 3.5 Flash and GLM-5 as executors and Claude Opus 4.8 as the default editor. Those are maxima across configurations. One practical note for anyone reproducing them: the released example environment leaves model IDs blank, so the editor falls back to the solver's model unless you set it.
Setup
We pinned the release at commit 6868817 and first ran its local demo in both recursion modes (28 Python executions, 32 scripted editor calls) and its focused test suite (17 tests and 39 subtests, all passing). The demo's editors are deterministic programs, so this exercises the workflow and leaves model behavior for the experiment.
The scenario is a production incident. A configuration change starts failing requests. The agent has to investigate, merge a repair, wait for the deploy and confirm that the service recovered. Partway through, a teammate posts a status request in the incident channel. Eight required checks read the agent's API calls and the resulting state of the twins, including whether it investigated the regression, deployed the repair, saw the monitor recover and answered the request thread. Two controls calibrate the checks: an agent that does nothing fails all eight, and a scripted reference passes all eight.
The agent under test is OpenCode running Claude Haiku 5.5, starting from the weakest version of the instructions in our on-call example. Its baseline passed seven of eight checks. It fixed the service and never noticed the question.
Harness² ran with its code unmodified. We restricted edits to prompts and skills, and Ancha's scores never entered the editor's or the composer's context. After each episode, a step outside Harness² selected the highest-scoring harness, with ties going to the composition. That selection rule is our addition to the paper's protocol.
Results
Table 1 shows the two optimization episodes. Both editors started from the same baseline run.
| Editor | General edit | Domain edit | Composed |
|---|---|---|---|
| Claude Sonnet 4.6 | 7 / 8 | 7 / 8 | 7 / 8 |
| Claude Opus 4.8 | 8 / 8 | 3 / 8 | 8 / 8 |
Sonnet's general edit added a rule for replying to requests. The agent answered the thread and stopped querying the error metric. Its domain edit added a status-reply skill, and the agent still missed the message. Knowing how to answer did not make it look for the question. The composer dropped a cached list of workspace IDs that would not transfer to another workspace, which was the right call, and then described the runs as successful. The checks disagreed. None of the three runs passed every required check.
Opus's general edit added one paragraph to the system prompt and a new skill.
harnesses/opus-selected/systemprompt.md+ A human will post a status request as a NEW message in the incident channel while + the incident is in progress; it is not there at the start, so re-read the channel as + you go and do not finish until you have checked for it. When a human asks for status + during an incident, or before you finish an incident, load the `status-request` skill. harnesses/opus-selected/skills/status-request/new skill
That probe passed all eight checks, and the composer kept it. The domain edit scored 3 of 8 after the agent misread the configuration and ran into a branch-name collision in the example's pull-request helper, a failure we also saw once from the unmodified baseline.
We then froze each selected harness and ran fresh incidents with new resource IDs, interleaved across harnesses. Nothing was edited after the freeze.
Scope: three fresh trials per harness on the delayed-request task, one trial on each guard task, and one optimization episode per editor. These results do not establish a general reliability rate. Reproduce the experiment from the code and data; see the method and limitations.
The Opus harness answered the status request in all three fresh incidents. One of those runs skipped the metric query, so two of three passed every check. On the bad-deploy and false-alarm scenarios, every harness passed.
What the edit learned
The winning edit is specific. It tells the agent that a status request will arrive as a new message after the incident starts. That is exactly what the probe trajectory showed, and it is what a careful engineer would write after reading the same trace.
It is also slower. On the bad-deploy scenario the Opus harness took 161 seconds against the baseline's 83, one run each, a cost that a pass/fail score does not show.
Fresh resource IDs test whether a fix survives new data. They do not test whether it survives different people: a teammate who never asks, who asks in a thread, or who asks after the deploy has finished. Telling a general habit from a memorized scenario takes worlds where the people vary as well as the data.
What the environment contributed
Harness² proposes changes; something has to judge them. Four properties of the environment did most of that work here.
- A fresh world per run. Every probe started from the same seeded state, so differences between runs came from the harness.
- Checks on consequences. Required checks read API calls and the twins' state. A confident final message could not pass a run that never answered the thread.
- Calibrated scenarios. The null and reference controls bound what each check can say, and infrastructure failures are recorded as invalid instead of being charged to the agent. None of the 22 model-driven runs was invalid.
- Scores outside the loop. The editors never saw a verdict, so the comparison measures Harness² as released, with selection layered on top.
Agents built on coding-agent harnesses and agent SDKs often reach their tools through MCP. Ancha 0.1.5 serves these four twins as MCP endpoints, configured with ancha mcp config. The runs in this post predate that release and used a small stdio bridge over the native twins. The bridge is in the repository.
$ npm i -g @ancha-labs/cli
Cost and scope
The whole study used $3.17 of OpenRouter credit, including setup and smoke runs. The executor was Claude Haiku 5.5, at $0.10 and $0.50 per million input and output tokens. The editors were Claude Sonnet 4.6 ($3 and $15) and Claude Opus 4.8 ($5 and $25). The Sonnet episode took about 20 minutes and the Opus episode about 16, reusing the same baseline run.
This is a case study: one scenario family, three fresh trials per harness on the hard task, one optimization episode per editor, and edits limited to prompts and skills. We did not rerun the paper's benchmarks, whose stock setup uses Vertex AI. The Harness² README also flags a risk worth keeping in mind when reading leaderboard gains: executors have open network access, so public answers or grading criteria may be reachable. In our runs the agent's tools pointed only at local twins, though we did not test a hardened network boundary.
Code and data
The adapter, the bridge, the scenario, all three harnesses, every score and the cost meter are in ancha-labs/harness2-eval.
Thanks to the Harness² authors for releasing a codebase that keeps every candidate, trace and decision on disk. It made this study easy to audit. Harness² is a Google Research repository, not an officially supported Google product. This is an independent study; it implies no endorsement of, or partnership with, Ancha.