Inside Harness2: Learning to Improve Without an Answer Key

A video about stories and uncertainty stayed with me while I was reading Harness2.

It explored how readily we give events the shape of a story: a setback, a turning point, a breakthrough. We can often make that shape convincing afterward. In the middle of an experience, its consequences are still unfolding.

“We don't know enough about life to know what the good news is and the bad news is.”

— Kurt Vonnegut, discussing Hamlet in his “Shape of Stories” lecture, February 4, 2004.

What stayed with me was how quickly we decide what an event means. A difficult experience can become a turning point in the story we tell afterward. At the time, its place in that story may be unclear. I think we can be similarly quick to give an agent experiment the shape of a breakthrough before examining what it actually changed.

What interested me was the possibility of moving forward without pretending to know how the whole story ends. Make a decision, observe what follows, and reconsider what the experience actually supports.

I found a useful engineering question in that idea: when we change an agent, what gives us a reason to call the change an improvement?

A prompt can sound more thoughtful. A workflow can have more stages. An answer can look more polished. Each makes it easy to tell a story about progress. What happened during execution might tell us something else.

Harness² gives that question a concrete structure. It changes the system around a model, runs the task again, examines the consequences, and retains supported experience for later refinement. The model weights stay fixed.

That is what I wanted to understand in this deep dive: how the repository moves from a plausible change to an observed effect, and from an observed effect to something worth carrying forward.

We will start with the moving parts, follow a small execution, and then open the relevant files. You need a basic understanding of language models and tools; familiarity with the repository is optional.

Reading note. Code links are pinned to repository commit 1cfe254. The ledger outputs below were checked with the repository's deterministic local demo. Benchmark numbers are the authors' reported results; I have not independently reproduced them.

1. What changes, and what carries forward?

An execution harness is the machinery through which a model does work: instructions, helper code, available tools, reusable skills, memory, and delegation. Changing this layer changes what an agent sees and can do without retraining its model.

Harness2 also has an improvement harness: the instructions and accumulated experience that guide the process of editing execution harnesses. That gives us two loops:

• Within one task: refine the execution harness using evidence from attempts at that task.
• Across tasks: retain useful harnesses and lessons so later refinement can draw on earlier experience.

The important implementation detail is where continuity lives. Each new task starts from the common base execution harness. Earlier work influences the refinement process through a library; the last task's harness does not silently become the next task's starting point. The task scheduler is a good place to verify that distinction.

The model stays fixed. Executable harness files change within the task; supported experience persists between tasks.

What can actually change?

For the OpenCode executor, the editable surface is concrete files:

The first four are standing instruction components; skills and subagents have their own activation mechanisms. The OpenCode surface definition specifies these paths. Codex has a separate mapping, so this table should not be treated as a universal directory contract.

These are different engineering interventions. Adding prose can guide a model; changing a script alters an actual computation. A plugin can affect the tool loop itself. Choosing the right component matters as much as choosing the right wording.

2. One refinement step: Propose, Probe, Compose

“Life can only be understood backwards, but it must be lived forwards.”

— Søren Kierkegaard, journal entry JJ:167 (1843), in a commonly shortened English rendering. Source and original wording.

That forward-and-backward relationship is the clearest connection I see between the video and Harness2. An edit is proposed before its consequences are known. Execution then gives the editor something to look back on: what the agent did, what it produced, and where its behavior differed. The next decision can draw on that record, even though it still comes with uncertainty.

Here is how the repository makes that process concrete.

Start by running the current harness on the task. Keep both the trajectory, meaning the actions and observations during execution, and the deliverable, meaning the resulting answer or artifact.

The editor then proposes two kinds of change:

• An invariant candidate addresses a general execution problem, such as weak planning or unreliable tool use.
• A domain candidate addresses procedures or constraints specific to the task's field.

These are two scopes for investigation. They can touch overlapping components and can even converge on the same fix. The stage prompts define what each editor should look for and what evidence composition should use.

Next, each candidate harness executes the same task. This is Probe. A proposal now has a behavioral consequence that can be compared with the reference execution.

Finally, Compose reads the candidates, their changes, and the execution evidence. It can keep supported changes, resolve conflicts, revise a proposal, or preserve the existing harness. The composed harness executes again to produce a deliverable. These stages are orchestrated in recursion.py.

A proposal describes what the editor hopes a change will do. A probe gives us an execution to examine.

Consider an illustrative document workflow. An instruction to check every citation sounds useful. In practice, it might produce repeated browsing while leaving the final document unchecked. Watching the execution exposes the gap between a sensible intention and its actual consequences.

The evidence is a comparison of behavior. Its strength depends on what was observed: a changed tool call, a corrected computation, a missing artifact, a new failure.

I find that a useful discipline for reading claims about agent improvement. Before accepting the explanation, I want to see the behavior it is explaining.

3. Follow a ledger through the workflow

The repository's local demo makes those abstractions visible. Its task asks for the sum of every signed amount in a CSV ledger. The first ledger contains:

100.25
-20.10
  5.00

The base harness includes a helper script that sums only positive amounts. It returns 105.25, dropping the negative entry. Including every signed amount gives 85.15.

The demonstration changes the helper's computation so negative entries participate in the sum. The candidate executions and composed execution then return 85.15. The bug, task data, and scripted editor are all visible in demo.py.

Here is the observed trace for one step, k=1, in parallel mode:

The demo uses deterministic editors and real Python executions. It shows how evidence, edits, and outputs travel through the machinery. It does not demonstrate a language model independently discovering this bug.

The second ledger is especially informative. Its amounts are 35.50, -10.25, and -5.00. Although an earlier library entry now exists, its base execution still returns 35.50. Refinement produces 20.25.

That behavior makes the two-loop design tangible: experience is available to the editor, while the task still starts from the common base. The reusable lesson is about including both signs in the calculation. The previous ledger's numerical answer has no value for the new ledger.

This example has a definite requirement, and we can check its arithmetic.

For this two-task, one-mode run, the demo recorded eight solver executions and eight editor calls. Per task, those are four executions plus two proposals, composition, and reflection. The demo's command implementation exposes these counters in demo.json.

4. How evidence reaches the editor

The editor needs more than a sentence saying an attempt failed. Harness2 builds a directory of evidence that the editing session can inspect. Conceptually, it contains:

context/
  task.md          task request
  source/          input materials, when available
  harness/         component files being edited
  rollouts/        execution trajectories
  deliverables/    produced artifacts or their available representation
  pool/            candidate harnesses and execution evidence
  library/         earlier relevant experience

The exact contents depend on the benchmark and stage. For example, an adapter can provide deliverable contents or only a file inventory. Individual observations can also be truncated. Storing evidence as files does not guarantee the editor will read every file.

context.py assembles this workspace and points the editor toward what is available. This organization makes the refinement inspectable: you can connect an edit to the source material, trajectory, and artifact used to justify it.

Proposed files also pass through structural checks. tree.py defines allowed paths and validates the component tree. Those checks can reject malformed harness edits. They cannot establish that a legal memo is accurate or that a generated application satisfies its user.

That distinction is useful whenever you inspect an agent system: successful execution, valid structure, and correct work are separate properties.

5. Wider search or deeper refinement?

The --k option controls the task-level budget. Both modes collect k base reference executions, then use that budget differently.

Parallel mode proposes k invariant and k domain candidates. After probing them, it runs k independent compositions, each able to inspect the shared candidate evidence.

Sequential mode performs k successive refinement steps. Each step uses the current harness and accumulated history, so later edits can build on or correct earlier ones.

These schedules are implemented in the two recursion functions. At k=1, both reduce to one contrastive step.

Width gives composition more alternatives. Depth gives later steps a changed starting point.

For k>1, a selector compares anonymized final deliverables with the task request and source materials. It chooses an existing result; it does not write a new answer. Sequential selection considers the composed outputs from every round, so it can adopt an earlier round. At k=1, selection is bypassed.

There is a consequential implementation choice here: the selection pool contains the composed finals. It excludes the original base executions and intermediate proposal probes. Composition can preserve the base harness, but the selector cannot directly choose an excluded base output. You can inspect this in selection_pool().

6. Keeping an answer, keeping a reason

After adoption, reflection examines the refinement record. A useful lesson needs a situation where it applies, a proposed practice, and evidence supporting that practice.

The library stores the adopted executable harness alongside an EXPERIENCE.md file. An index lets later editing sessions find relevant entries. The retention code requires nonempty cue, lesson, and evidence fields before adding an entry; a selected answer alone is insufficient. See library.py.

In the ledger example, an appropriate lesson concerns signed aggregation. A poor lesson would memorize 85.15, or claim every future accounting failure can be fixed by deleting a filter.

This is the part of the architecture that stayed with me most: the difference between keeping an answer and keeping a reason.

The answer belongs to that ledger. The reason may be useful elsewhere, if the conditions still apply. A reusable artifact carries how to act; its experience entry carries when and why that action helped.

Looking back can make an attempt understandable. It can also tempt us to make its lesson too broad. Keeping the situation and supporting evidence alongside the lesson gives a future editing session something concrete to adapt and something concrete to question.

The video's television-series analogy helps me think about that limit. A resolution in one episode can create a problem in the next. Similarly, a change that helps one task may bring assumptions that fit another task poorly. Retaining the conditions behind a lesson gives a later editor a basis for deciding whether it applies. It avoids treating one task's successful ending as the ending of the whole story.

The release organizes these libraries by domain. Tasks within a domain stay ordered and read earlier entries; different domains can run concurrently. Run settings separate library namespaces. The experiment guide explains that scope. Cross-domain reuse should therefore be treated as an explicit experiment, rather than assumed from the presence of a library.

7. What do the results establish?

The paper evaluates three professional benchmarks with GLM-5 and Gemini 3.5 Flash under OpenCode and Codex. For a single refinement step, it reports maximum gains over the corresponding base harness of:

These maxima come from different configurations. They are not an expected improvement for every task. Main tables report means and standard deviations over three runs with shuffled task order. Additional parallel recursion at k=5 brings further gains in the reported experiments. See the paper's experimental setup and main results.

Those findings support testing harness refinement as an engineering approach. Whether it pays off in another workload depends on the task, executor, editor, budget, and evaluation.

Account for the full workflow

The code schedules 4k solver executions and 3k improvement stages for one task in one mode, including the base references. Reflection is additional, as is selection when needed. Retries and the work inside each session affect actual usage. The execution ledger verifies these stage counts; its dollar fields are not a complete measure of end-to-end solver cost.

A shorter adopted trajectory can coexist with more total compute spent finding it. I would compare end-to-end cost and latency alongside deliverable quality before choosing a recursion budget.

In an agent system, iteration needs a budget and a purpose. More activity can introduce regressions, spend money, or leave the important failure untouched. Sometimes the evidence supports keeping the existing harness or adopting an earlier composed result.

Keep refinement and evaluation separate

The refinement process uses task evidence without ground-truth grades. Benchmark judging happens separately after results exist; judge.py handles that evaluation layer.

This separation makes the setup relevant to work where no answer key is available during execution. It also leaves a reliability limit: an editor and selector can prefer an artifact that still contains a substantive mistake. Independent verification remains useful whenever the task permits it.

There is also a concrete evaluation concern in the release: executor network access is unrestricted, and public grading material can be reachable online. The authors document a validation rollout retrieving a rubric. Reproductions need explicit network restrictions and should report them. The web-access notes explain the issue.

8. Try the smallest useful experiment

Start with the deterministic demo. From a checkout of the pinned revision, the documented installation uses Linux, Python 3.12, and uv:

git clone https://github.com/google-research/harness2.git
cd harness2
git checkout 1cfe254e8d25b15f37048c3730134686abe60335
uv venv --python 3.12 .venv
source .venv/bin/activate
uv pip install -e '.[render]'
harness2-demo --output runs/demo-k1 --k 1 --mode parallel

The demo needs no model credentials. Inspect its demo.json, execution outputs, and retained experience together. That connects the terminal result to the files that changed.

For model-backed benchmarks, use the setup guide. It covers model configuration, authentication, Node.js, and benchmark dependencies. Preflight checks do not guarantee model availability; calls test that when they occur.

A small LAB run, after setup and authentication, can follow this sequence:

export HARNESS2_RUNS_DIR=runs/lab-first-task
harness2-setup lab
harness2-check --bench lab
harness2 --bench lab --k 1 --mode parallel \
  --only-domains antitrust-competition --max-tasks 1 --dry-run

Use a fresh output directory for this experiment. After reviewing the selection and configuring evaluation network restrictions, remove --dry-run to execute. Model-backed runs incur API costs. Keep the same output directory and settings when judging and reporting:

harness2-judge --bench lab --k 1 --mode parallel
harness2-report --bench lab --k 1 --mode parallel

One easy-to-miss detail: --max-tasks caps tasks per domain. Restricting the domain here keeps the example small. Judging reads saved task summaries from the output namespace; it does not reapply the refinement command's task filters. A populated namespace can therefore include earlier tasks. See the judge command and experiment instructions.

9. Read the repository in this order

The code separates three concerns: the refinement method, the execution framework, and the benchmark adapter. The BenchmarkAdapter interface describes how a benchmark supplies tasks, source materials, execution, and deliverables to the shared machinery.

Use these questions to guide a first pass:

What I want to carry forward

What interests me about Harness2 is a way to make the next attempt inspectable. It gives us a concrete change, an execution to examine, and a record of why the change might be worth reusing. Each part can be questioned.

That is the discipline I want to bring into my own agent work: change a component, examine its consequences, and keep a lesson only as far as the evidence supports it. Let the next task challenge how far that lesson travels.

We can make a thoughtful next move without claiming to know the ending. We can also recognize a small improvement when we have evidence for one: a calculation now includes every input, an artifact meets a requirement, or a failure becomes easier to explain.

That brings me back to the video's closing thought. A corrected ledger calculation establishes something small and definite. I want to be able to appreciate that result while staying curious about what the next task will reveal.

“If this isn't nice, I don't know what is.”

— Alex Vonnegut, as recalled by his nephew Kurt Vonnegut in his 2004 Lehigh University commencement address.

Sources and inspiration. I was reading about Kierkegaard and Vonnegut while exploring Harness2, and their thoughts on uncertainty stayed with me. Technical analysis is based on the Harness2 repository and Xu et al., Harness²: Recursive Agent Harnessing for an Open World. Explanations and diagrams here are original. Harness2 is a research release, not an officially supported Google product.