TL;DR
- Eight editable harness components, one Propose-Probe-Compose step, model weights frozen throughout
- GLM-5 with a refined harness scores 75.7 on WorkBuddy-Bench, above Gemini 3.5 Flash base at 72.4
- 9,275 stars and 74 open issues against three commits since 8 October
System Requirements
| RAM | No GPU needed for the demo; Linux with Python 3.12 and uv |
| GPU | None, the agents run against hosted model APIs |
| VRAM | Benchmarks need a Vertex AI project and the calls are billed |
Google Cloud AI Research published a framework that improves an agent by rewriting everything around the model and never touching the weights. The headline number is 11.5 points. The number underneath it is better: a weaker model with a refined harness outscores a stronger model with its default one. That is the result worth taking back to your own stack, and you can read the whole thing because Harness2 shipped under Apache 2.0.
What Harness2 actually does
A harness is everything wrapped around a frozen model: the system prompt, the memory, the tool descriptions, the subagents, the rules it must obey. Every agent you run has one, whether you designed it or inherited it from a default config. The field has spent two years arguing about models and shipping harnesses by feel.
Harness2 treats that layer as the thing to optimise. It defines eight editable components and lets an editor model rewrite them while the task is running:
| Component | File in the harness tree | When it costs context |
|---|
| System prompt | the agent's standing instructions | Every call |
| Guardrails | guardrails.md | Every call |
| Memory | LongTermMEMORY.md | Every call |
| Tool guidance | tool_guidance.md | Every call |
| Subagents | sub_agents/<n>/AGENT.md | Only when the task tool fires |
| Skills | skills/<n>/SKILL.md | Only when invoked |
| Scripts | executable helpers | Only when run |
| Plugin | plugin/<name>.ts | Middleware, inside the loop |
That table is itself an argument, and it is the most portable thing in the paper. Four components sit in context on every single call and four cost nothing until they are needed. The repo's own guidance is blunt about the consequence: always-on prose is the expensive option, because every word competes with the agent's output budget. If you write nothing else down from this release, write that down.
The loop is called Propose, Probe, Compose. An editor proposes two kinds of edit, one general and one specific to the task's domain. Each candidate is probed by actually running the task again. A composer merges the parts that held up into a single harness. Then a reflector files what was learned into a library that the next task reads from.
The part that makes this usable in the real world is that none of it needs an answer key. The edits are chosen by contrasting how the runs behaved, not by scoring them against ground truth, because in an open world you do not have ground truth at the moment you need it.
The headline number, and the better one underneath it
Here is WorkBuddy-Bench with OpenCode as the starting harness, from Table 1 of the paper. Scores are macro-averages across the code, web and office domains, with standard deviation over runs.
| Executor | Method | Avg. score | Avg. steps |
|---|
| Gemini 3.5 Flash | OpenCode base | 72.4 | 25.4 |
| Gemini 3.5 Flash | ReasoningBank | 73.4 | 24.9 |
| Gemini 3.5 Flash | Meta-Harness (offline) | 76.4 | 28.8 |
| Gemini 3.5 Flash | Harness2 | 77.5 | 21.9 |
| GLM-5 | OpenCode base | 64.2 | 13.5 |
| GLM-5 | ReasoningBank | 65.6 | 14.4 |
| GLM-5 | Meta-Harness (offline) | 69.7 | 15.3 |
| GLM-5 | Harness2 | 75.7 | 13.7 |
The 11.5 points everyone is quoting is GLM-5 going from 64.2 to 75.7. Real, and the largest single-step gain on the benchmark.
Now read the two bold rows against each other. GLM-5 with a refined harness scores 75.7. Gemini 3.5 Flash with its default harness scores 72.4. The cheaper, weaker executor wins by three and a bit points, purely because somebody rewrote the scaffolding. The paper states this directly: weaker executors benefit more. If you have ever picked a model upgrade over a day spent fixing your prompts and tool descriptions, that comparison is an argument you should sit with.
Look at the steps column too. Gemini drops from 25.4 steps to 21.9 while scoring five points higher. Better and shorter, not better and slower.
Three of the four baselines make things worse
This is the section most coverage skipped, and it is the most useful one if you are choosing an approach rather than reading a press release.
- ReasoningBank, which keeps the harness fixed and retrieves past strategies into context, falls as much as 6.6 points below the base harness on coding tasks.
- Harness Scaling, which revises per instance without carrying state, underperforms the base harness in seven of twelve benchmark and configuration pairs.
- Meta-Harness, which searches offline against a training split, is the only genuine rival. It still beats Harness2 on LAB partial pass rate for Gemini under Codex.
So the finding is not "scaffolding tweaks help." Most scaffolding tweaks hurt. Adding retrieved context to a working agent made it worse on code. That matches what anyone who has over-stuffed a system prompt already suspects, and it is nice to see it measured.
The ablations back the design up. Removing the cross-task library costs 3.1 points on WorkBuddy-Bench and 1.8 on JobBench. Removing the scoped editor roles costs 2.9 and 3.7. Removing the composition step costs 1.6 and 1.9. All three parts carry weight, and the scoping, which is the cheapest part to implement yourself, is the one carrying the most on JobBench.
Google's framework uses Anthropic's model as the editor
From the paper's implementation details, stated plainly: "The default harness editor is Claude Opus 4.8 (Anthropic, 2026) running under OpenCode."
Google Cloud AI Research published an Apache 2.0 agent-improvement framework whose proposing brain is a closed, paid, competitor API, driven through OpenCode, an MIT community harness. The repo's setup docs keep it configurable, with HARNESS2_IMPROVER_MODEL accepting either Vertex Claude or Gemini and defaulting to whatever the executor is, so you are not locked in. But the numbers in the paper were produced with Opus doing the editing.
Nobody is hiding this. It is in the paper. It is just a strange shape for the field to have arrived at, and it tells you something about which model people reach for when the job is reading a trajectory and writing better instructions.
The executors, for the record, are Gemini 3.5 Flash and GLM-5, under OpenCode and Codex. Two models, two harnesses, four configurations, three benchmarks.
What it costs, honestly
Probing means re-running the task. Composition means another editor session. That is real money and real wall-clock, and the paper does not hide from it: the compute and performance trade-off plot includes both execution and improvement costs, and Harness2 configurations sit on the empirical Pareto frontier. Meta-Harness uses more compute at k=5 for lower performance.
Pushing the recursion budget from k=1 to k=5 with Gemini under OpenCode takes the gain over base from 4.5 to 7.5 points on JobBench and from 5.1 to 8.4 on WorkBuddy-Bench. Diminishing, but still climbing. The paper's own limitations section says parallel sessions reduce latency when you have the capacity but do not remove the extra computation, and that their experiments map a bounded range of budgets rather than finding an optimum.
The second stated limitation is the one to take seriously. Without ground-truth supervision, refinement leans on behavioural contrast and model judgment, which are sensitive to execution variability and to shared model biases. An adopted change is not guaranteed to be correct and is not guaranteed to transfer. You are letting a model decide which version of its own instructions worked better.
About the benchmark-leakage warning
The README tells you to configure network restrictions before evaluating, because public benchmark rubrics may be reachable online, and Codex runs with its sandbox disabled. That reads alarming out of context, so here is the fair version.
The framework is designed against leakage internally: the docs state that rubrics are kept away from the improver on JobBench, and that the improver never receives verifier output on WorkBuddy-Bench. The warning is about the executor having unrestricted web access and being able to fetch a public RUBRICS.json during a run. That is a real hazard for anyone reproducing the numbers on their own box, and Google flagging it in their own README is the opposite of hiding it. Block the hosts, as they say to.
Run the demo in under ten minutes
The local demo needs no cloud credentials and no GPU, which makes this unusually cheap to form an opinion about. Linux, Python 3.12 and uv:
uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install -e '.[render]'
harness2-demo --output runs/demo --k 2
Real benchmark runs are a different story. You need a Google Cloud project with Vertex AI and the calls are billed, plus Node.js 20+, Docker for WorkBuddy-Bench, and Pandoc and Poppler for LAB. The full loop on the legal benchmark:
harness2-setup lab
harness2-check --bench lab
harness2 --bench lab --k 1 --mode parallel \
--only-domains antitrust-competition --max-tasks 1
harness2-judge --bench lab --k 1 --mode parallel
harness2-report --bench lab --k 1 --mode parallel
Judging and reporting must use the same benchmark, executor, subset, --k, --mode and seed as the refinement run, or the report is meaningless. Start with --max-tasks 1.
Note also that WorkBuddy-Bench carries a custom Tencent licence with a territorial limitation. The Apache 2.0 on the framework does not extend to the benchmarks it downloads.
What is actually in the repository
Here is the part that keeps this honest. As of today the repo has 9,275 stars, 818 forks and 74 open issues, against three commits from two contributors. Initial commit on 8 October, an "updates" commit on 9 October, and on 14 October a commit titled "Fix arXiv link." Nothing since. No releases, ever.
Seventy-four issues filed against three commits is a precise measurement of how the field now treats a Google paper drop: thousands of people star it, hundreds fork it, dozens hit problems, and the authors have moved on. The README says plainly that this is a research project and not an officially supported Google product, and that it is not eligible for Google's vulnerability rewards programme. Believe them.
That does not make the work less good. It does mean you should treat Harness2 as a published method with a reference implementation, not as a dependency. Read the loop, steal the eight-component model, implement the scoping in your own stack. Do not put harness2 in a production requirements file and expect someone to answer your issue.
Who should use it
Read it today if you maintain an agent harness of any kind. The eight-component taxonomy and the always-on versus on-demand distinction are free, apply to every harness in our harness wars coverage, and will make your system prompt shorter this afternoon.
Run it today if you have a Vertex AI budget and a benchmark you actually care about. The demo is free; the real runs are not.
Skip it if you wanted a maintained library. Three commits and no releases is not a product, and the multi-agent orchestration tools and memory layers we have covered are further along as software even where they are behind as research.
One last thing worth saying out loud. The paper frames this as recursive self-improvement, and the weights are frozen throughout. Nothing here learns. It is a search over scaffolding, with a large model as the proposal function and a re-run of the task as the test. That is a genuinely good idea and it does not need the bigger label, which is why it is a little disappointing to see the label applied.
Sources and further reading
Your under-ten-minutes move: open your own agent's system prompt and sort every line into always-on or on-demand. Everything in the first group is being paid for on every single call. Harness2's own guidance says prefer the second group wherever a rule can be a check instead of a paragraph, and you do not need a Vertex project to act on that.
Tested on: not independently tested. Every benchmark figure is from the Harness2 paper (arXiv:2610.14224 v1, 9 October 2026) and was produced by its authors. Repository statistics were read from the GitHub API on the date below.
Date checked: 2026-10-19
