Harness²: Can an agent improve its workflow without a correctness oracle?

Review date: 2026-10-24 · Review author: Zhongzhu Zhou

Paper: Harness^2: Recursive Agent Harnessing for an Open World. Paper authors: Ruiyao Xu, Yanfei Chen, Zhongying CuiZhu, Bhavana Dalvi Mishra, Yifei Ming, Han Yu, Rujun Han, Chen-Yu Lee, and Tomas Pfister. Version: arXiv:2610.14224v1, submitted 2026-10-09. Sources: paper and metadata, full PDF, project, and official resource repository. This review covers the paper and all appendices. Numerical results below are the authors’ reported measurements; analytical examples and proposed experiments are identified separately.

1. The central question: what can improve when the weights do not?

An agent can fail with a capable language model because its surrounding workflow makes the wrong behavior easy. It may repeatedly read irrelevant files, lose important evidence in tool output, choose the wrong aggregation grain in a spreadsheet, or spend its entire budget explaining what it intends to deliver. These are not all failures of factual knowledge. Some are failures in how instructions, tools, memory, and work allocation meet the model at execution time.

Harness² treats that surrounding system as editable. Its most interesting question is not whether a longer prompt improves a benchmark. It asks how to change an execution harness on an unfamiliar task when there is no trusted answer key, and how to preserve useful experience from that change for later tasks. Those two questions correspond to two different update timescales. Confusing them makes the method look either simpler or more autonomous than it is.

For the current task, the system proposes changes, executes the changed harnesses, and compares what happened. After completing the task, it stores reusable components and lessons about their effects. The next task still starts from the same base execution harness. What carries forward is an improvement library that informs new edits, rather than the preceding task’s entire customized working setup.

That reset is a substantive design decision. A rule useful for one legal document or one repository can become an incorrect assumption in another. Resetting reduces this direct inheritance risk while retaining the possibility of reusing a procedure. It does not make the library harmless: a retrieved lesson can still import a bad premise, and the editor can still mistake an accidental improvement for a general rule.

My reading is that the paper provides evidence for execution-informed workflow search with accumulated experience. It does not establish a correctness oracle, unbounded recursive capability growth, or an autonomous optimizer that rewrites every aspect of its own search policy. The model weights remain frozen; within a task, the improvement harness remains fixed; across tasks, the persistent update is implemented through its library.

Figure 1. Author-drawn schematic based on Sections 2.1–2.3. The return arrow denotes lessons becoming available to later tasks, not a change to the improver during the current task.

2. Prerequisites: model, harness, execution, and evidence

2.1 An agent is a model inside an operating procedure

Let $M$ be a frozen model, $q$ a task, and $H$ an execution harness. The agent induces a distribution over executions:

$$e=(\tau,y)\sim A(q;M,H).$$

The trajectory $\tau$ records actions, tool calls, intermediate observations, and decisions; $y$ is the final deliverable. Two runs with the same $M$, $H$, and task may differ. Sampling, tool behavior, and an agent’s contingent decisions mean that one successful run is not a deterministic property of the harness.

This distinction matters because the method uses both trajectories and outputs. Output-only selection can tell us which answer looks better to a judge. Trajectory inspection can suggest a mechanism: a new script prevented repeated hand calculations, a skill caused missing source material to be inspected, or a warning arrived before the output budget was exhausted. Mechanistic evidence is more actionable than a preference, but still falls short of proving that the edit caused the final gain.

2.2 Eight components differ in when they affect execution

The execution harness exposes eight editable components. System prompt, guardrails, memory, and tool guidance enter the model’s context on every call. Skills provide procedures loaded when triggered. Scripts package deterministic operations that can be invoked through the shell. Subagents permit bounded work in a separate context. Plugins operate inside the tool loop, outside the model’s choice of whether to read a prose instruction.

The distinction between these forms is an attention and reliability decision. A long procedure copied into always-on context has a recurring token cost and competes with the current task. A skill can remain dormant until needed, but an untriggered skill has no effect. A script can make repeated arithmetic consistent, but consistency is harmful if it encodes the wrong formula. A plugin can enforce an event more reliably than advice, but it can also constrain execution in the wrong situation.

The paper’s appendix makes activation dependencies explicit. A skill needs an invocation cue; a referenced helper must accompany it; malformed harness trees receive validation feedback and one repair attempt. If repair fails, the previous harness is inherited. These checks establish that a proposal can be mounted and used. They do not establish that its domain assumptions are true.

Figure 2. Author-drawn grouping of Figure 2 and Appendix D.3. The grouping emphasizes activation and scope rather than directory names.

2.3 Behavioral contrast is weaker than ground truth

Suppose a base agent computes an average of row-level commission rates. An edited agent instead sums commission and divides by total sales. If the request requires a sales-weighted rate, the edit addresses a clear defect. The trajectory reveals a meaningful change in reasoning and execution. If the request instead asks for an unweighted average over representatives, the very same edit is wrong.

The requirement and the source material are therefore part of the evidence, not optional context. A successful tool call, a well-formed artifact, and agreement between two models are all compatible with a wrong answer. In Harness², the improver may inspect the request and source materials but not hidden benchmark grading criteria or reference answers. “No ground-truth verifier” means that the adaptation loop lacks that privileged feedback; it does not mean the evaluation has no grader.

For later analysis, it is useful to imagine a hidden quality function $G(q,y)$. This is an explanatory device used in this review, not an objective that the deployed editor can evaluate. The system acts on observable evidence while the experiment assesses outcomes using information unavailable during adaptation. Keeping those two views separate is the key to reading both the gains and the failures correctly.

3. One improvement step: propose, probe, compose

The paper defines a contrastive operation:

$$(H',e',\xi)=\Phi(q_t,H,E;R_t).$$

Here $E$ contains the execution and improvement history available within task $t$. The improvement harness $R_t$ includes the library that guides editing and composition. The returned record $\xi$ contains the reference executions, candidate edits, probes, composition decisions, and successor execution. The successor is actually run; it is not merely a promising plan saved without observing its consequences.

3.1 Two proposal scopes

The invariant editor targets problems in the agent’s general operating procedure: planning, tool use, context management, delegation, or budgeting. The domain editor targets the practitioner knowledge and constraints needed for the current deliverable. Both read the current evidence, but their responsibilities differ.

This separation is designed to diversify diagnoses. If two unrestricted editors both decide that the agent needs to “be more careful,” the system gets little additional information. A process editor might instead notice repeated parsing mistakes, while a domain editor notices that a reconciliation must occur at the transaction level. Their edits can complement each other: a script handles parsing, and a skill specifies what must be reconciled.

The split is an inductive bias, not a theorem about independent error sources. Domain knowledge often changes reading order, and a workflow constraint can encode a domain assumption. Composition must resolve overlap instead of concatenating both proposals and hoping that more instructions help.

3.2 Probing observes changed behavior

Each candidate runs on the same task under matched model and environment configurations. Separate fresh containers prevent one rollout’s modified workspace from becoming another rollout’s starting state. The reference execution supplies a baseline behavior, not a correct reference solution.

The most useful probe shows where the intended mechanism enters the trajectory. Did the model load the skill? Did the script consume the correct source? Did the revised guardrail cause an omitted file to be examined? Did the intervention introduce new retries or prevent completion? Those observations support a narrower conclusion than “the new harness is better,” but they are more useful for deciding which edit to preserve.

Matched configurations do not remove stochasticity. One candidate execution can be lucky, and two editors can share the same misconception. The paper explicitly acknowledges both. The reflection prompt also distinguishes helped, hurt, no observed effect, and confounded attribution. That last category is essential when several components change at once.

3.3 Composition is a new proposal that must be executed

The composer sees the candidates, their diffs, and all available execution evidence. It may combine supported changes, revise conflicts, adopt one candidate, or keep the current harness. Keeping the current state is a legitimate outcome when the evidence does not support a change.

Merging is not equivalent to averaging scores. Two useful changes can interfere through context size, trigger order, or incompatible requirements. The composed harness therefore receives its own execution. This extra run is part of both the method’s evidential value and its cost.

Algorithm 1. Basic contrastive improvement

The pseudocode below summarizes Section 2.2.1; it does not introduce access to a hidden grader.

Input: task q, current harness H, history E, fixed improver R
1. Execute H on q to obtain reference e0 = (trajectory, output).
2. Propose an invariant candidate HI using q, H, e0, E, R.
3. Propose a domain candidate HD using the same evidence.
4. Validate each harness tree; repair once or inherit H.
5. Probe HI and HD on q in separate fresh environments.
6. Inspect reference, edits, probes, and earlier task history.
7. Compose Hnew: merge, revise, choose one, or keep H.
8. Execute Hnew to obtain enew; record decisions and evidence.
9. Return Hnew, enew, and the complete step record.

Notice what is absent from this algorithm: there is no instruction to maximize a visible benchmark score. A general-purpose agent still makes the editing and composition decisions. The procedure improves the evidence presented to that agent, rather than replacing judgment with a verified optimizer.

4. Recursion can mean width or depth

The budget parameter $k$ has different operational meanings in the two variants. In parallel recursion, it specifies a population width: $k$ reference executions, $k$ invariant candidates, $k$ domain candidates, and $k$ independent compositions. Every composer receives the full candidate pool. The experiment performs one population update; increasing $k$ does not create $k$ successive generations of harness evolution.

Writing the reference pool as $B$ and the probed candidate pool as $P$, the paper’s parallel update is:

$$H'_j=\operatorname{Compose}_j(q_t,H,B,P;R_t), \qquad j=1,\ldots,k.$$

Sequential recursion instead feeds each refined harness into the next step:

$$\begin{aligned} H^{(0)}&=H_0,\\ (H^{(r)},e^{(r)},\xi^{(r)}) &=\Phi(q_t,H^{(r-1)},E^{(r-1)};R_t),\\ E^{(r)}&=E^{(r-1)}\cup\{\xi^{(r)}\}, \qquad r=1,\ldots,k. \end{aligned}$$

The history update is set notation for accumulating records, not an instruction to discard their order. The initial history includes $k$ base executions. Later steps can use both earlier proposals and the latest checkpoint’s behavior. A previously rejected idea can become useful after another defect is fixed, but an early mistaken assumption can also persist through the chain.

Figure 3. Author-drawn schematic of Section 2.2.2. Parallel composers share evidence; their independent sessions are not independent ground-truth witnesses.

Algorithm 2. Two ways to spend task-local budget

This summary separates harness search from the terminal selection step.

Input: task q, base H0, fixed improver R, budget k
1. Collect k base executions in fresh environments.
2. Parallel mode:
   a. Propose and probe k INV and k DOM harnesses from H0.
   b. Give the full pool to k independent composers.
   c. Execute the k composed successor harnesses.
3. Sequential mode:
   a. Initialize H = H0 and history with the base executions.
   b. Repeat k times: apply the contrastive step to H/history.
   c. Carry the successor harness and recorded evidence forward.
4. If k = 1, return the basic step's executed successor.
5. Otherwise anonymize eligible existing deliverables.
6. Select one using the request and source materials.
7. Return that existing deliverable and its associated harness.

For $k>1$, the selector sees anonymized deliverables without their originating harnesses. It chooses an existing output rather than synthesizing a new answer. This reduces a potential preference for an elaborately modified harness, but anonymization cannot eliminate a preference for an apparently polished or concise artifact. Appendix H even tells the selector that an unmodified-harness output is a valid choice. That is a useful protection against an assumption that adaptation must always win.

A transparent cost derivation

The parallel execution count can be derived directly from the procedure. There are $k$ reference executions, $2k$ candidate probes, and $k$ composed-harness executions:

$$N_{\mathrm{executor}}=k+2k+k=4k.$$

Thus $k=1$ already means four executor rollouts in the idealized complete schedule, and $k=5$ means twenty. This count excludes editor, composer, selector, reflector, validation-repair, and infrastructure costs. It is an accounting derivation from the described algorithm, not an empirical timing measurement. A fallback or a shared artifact can change incidental operations; variable rollout lengths mean twenty executions need not cost exactly five times four executions.

A fuller accounting expression is:

$$\begin{aligned} C_{\mathrm{task}}={}&\sum_{b=1}^{k} C_{\mathrm{base},b} +\sum_{p=1}^{2k} C_{\mathrm{probe},p} +\sum_{j=1}^{k} C_{\mathrm{successor},j}\\ &+C_{\mathrm{edit}}+C_{\mathrm{compose}} +C_{\mathrm{select}}+C_{\mathrm{reflect}}. \end{aligned}$$

The relevant billing boundary should say whether reflection, performed after returning the deliverable, is charged to the current task or to library construction. The paper’s compute–performance plot counts tokens spent before the final deliverable and includes execution and improvement costs. That is a much broader measure than the table’s executor step count, but it is not a substitute for a deployment-specific accounting of the whole stream.

Parallel execution can shorten the critical path if enough resources are available. It does not remove token expenditure. Sequential refinement offers the chance to repair defects discovered in an earlier edit, but introduces serial dependence. The paper’s budget curves favor parallel recursion at $k=5$ on the illustrated Code and Office domains; sequential recursion is stronger on Code at $k=2$. There is no universal ordering across budgets.

5. What the improvement library actually learns

After the task, the reflector examines the complete record, including proposals that were not adopted. This is important because selection operates on deliverables. An adopted harness may contain a useless edit, and a rejected candidate may contain the only edit that successfully fixed a specific behavior.

The library stores adopted harnesses together with experience about applicability, behavioral consequences, and supporting contrasts. The next editor can copy or adapt components, including dependencies, rather than merely reread a success story. The method therefore combines executable material with an explanation of when that material might help.

In abstract form, the cross-task update is:

$$\begin{aligned} L_{t+1}&=\operatorname{UpdateLibrary}(L_t,H_t^\star,D_t),\\ R_{t+1}&=\operatorname{WithLibrary}(R_t,L_{t+1}),\\ H_{t+1}^{(0)}&=H_0. \end{aligned}$$

Here $D_t$ is the full improvement record. The star marks the adopted harness, not a verified optimum. These equations are a compact restatement of Section 2.3. They expose the most easily missed fact: the execution harness resets while the material available to the improver accumulates.

Algorithm 3. Conservative cross-task retention

The summary preserves the appendix’s distinction between observation and attribution.

Input: adopted harness Hstar and complete improvement record D
1. Compare every proposed edit with the corresponding base tree.
2. Link each edit to the point where observed behavior differed.
3. Mark attribution: helped, hurt, no effect, or confounded.
4. Separate reusable procedures from facts unique to this task.
5. Write at most one durable lesson if the contrast supports it.
6. If support is insufficient, retain the record but add no lesson.
7. Store adopted components and supported experience in the library.
8. For a later task, reset to H0 and recheck any reused task facts.

The library is not simply a growing system prompt. Its usefulness depends on retrieval and selective application. The same distinction appears in everyday learning: remembering “check the transaction identifier before joining tables” is useful; remembering the previous client’s transaction identifiers as if they were universal is dangerous.

There is also an identification problem. A library entry contains both a concrete harness and a textual lesson. If later performance improves, which part mattered? The paper’s library ablation removes the combined mechanism. It supports the value of retained material as a package, but does not isolate learning a better editing strategy from retrieving a better component template.

6. Evaluation protocol before headline results

The main experiments cross two executor models, Gemini 3.5 Flash and GLM-5, with OpenCode and Codex initial harnesses. The main improvement agents use Claude Opus 4.8. The paper also studies settings where the same model performs editing and execution, which helps test whether a stronger editor is the only source of gains.

The domains are deliberately heterogeneous. WorkBuddy-Bench contains Code, Web, and Office work; the complete collection has 80, 70, and 50 tasks respectively, split in half by category. Reported test results therefore use 40 Code, 35 Web, and 25 Office tasks. The main average gives each domain equal weight. It is not the average over all 100 test tasks.

JobBench uses a frozen 31/34 train/test split of 65 tasks. LAB has 1,251 legal tasks in the source collection; the experiment evaluates a stratified set of 100 tasks and uses a separate 24-task development set for the offline Meta-Harness baseline. Its Partial metric is the fraction of criteria passed. It must not be read as the fraction of complete legal tasks that are wholly correct.

The training portions support offline Meta-Harness tuning. Other methods start with empty libraries on the test stream. Online library updates can use earlier test-task experience, so evaluation concerns continual adaptation within a stream, not independent identically initialized test examples. The three reported runs use independently shuffled task orders. Their standard deviations mix execution and order effects; they are not automatically confidence intervals over the entire real-world task population.

WorkBuddy combines deterministic checks with model or visual judges; JobBench uses weighted rubric evaluation; LAB uses criterion-level legal judging. These graders belong to the evaluation, not the editing loop. The separation is a major strength of the setup, but the results remain sensitive to how a grader represents a deliverable. A PDF’s visually correct content may not be accessible to a particular evaluator, and a legal answer can trade broad coverage for local precision.

7. What improves, and what the averages conceal

Table 1 reports WorkBuddy macro-averages of 72.4, 74.5, 64.2, and 64.4 for the four base configurations. One Harness² step raises them to 77.5, 77.5, 75.7, and 75.3. Adding the reported parallel-recursion increments gives 80.8, 80.4, 79.6, and 78.3 at $k=5$. Figure 4 redraws those means without inventing error bars for the derived $k=5$ values.

Figure 4. Redrawn from Table 1. k=5 values are calculated by adding each reported recursion increment to its k=1 mean; they are not new measurements.

The largest one-step WorkBuddy gain is $75.7-64.2=11.5$ points for GLM-5 under OpenCode. The largest JobBench one-step gain is $36.0-31.5=4.5$ points for Gemini under OpenCode. The largest LAB one-step gain is $71.4-59.6=11.8$ percentage points for GLM-5 under OpenCode. These maxima refer to different benchmarks and configurations; they should not be combined into a single average gain.

One can derive the distinction between a macro-average and a task-weighted average using Gemini/OpenCode domain means. The reported domain scores are 76.5, 76.4, and 79.5:

$$\begin{aligned} S_{\mathrm{macro}} &=(76.5+76.4+79.5)/3=77.47\approx77.5,\\ S_{\mathrm{weighted}} &=(40\cdot76.5+35\cdot76.4+25\cdot79.5)/100\\ &=77.215. \end{aligned}$$

The second number is a calculation from rounded means, not an additional paper result. It shows why task counts must accompany the metric. Neither aggregation is inherently wrong; each answers a different question about how much weight each domain should receive.

The paper’s broad pattern is robust across the evaluated configurations, but not every cell favors Harness² over every baseline. Meta-Harness is stronger on LAB for Gemini under Codex at the one-step setting. On the Code domain with GLM-5/Codex, Meta-Harness also exceeds the one-step Harness² score even though Harness² wins the three-domain average. These exceptions help prevent a misleading “best everywhere” reading.

Recursion does not uniformly erase differences between starting harnesses either. On WorkBuddy, the GLM-5 OpenCode–Codex gap changes from 0.2 at $k=1$ to 1.3 at $k=5$. A method can improve both configurations while widening their difference. That observation is compatible with better average performance; it simply limits the claim that adaptation makes the initial harness irrelevant.

8. Ablations: useful ingredients, incomplete causal separation

Table 3 removes the library, scoped editing roles, and composition separately from the $k=1$ system. The full scores are 77.5 on WorkBuddy and 36.0 on JobBench. Without the library they become 74.4 and 34.2; without role scoping, 74.6 and 32.3; without composition, 75.9 and 34.1.

Figure 5. Differences calculated from Table 3. Each bar compares a separately ablated system with the full system; the bars are not additive contributions.

The scoping ablation is particularly informative: it replaces the invariant and domain editors with two generic editors while retaining the number of proposals. This weakens the explanation that gains come merely from asking twice. It supports the value of differentiated search responsibilities. It does not prove that the chosen division is optimal or that the editors remain perfectly within their scopes.

The composition ablation retains two probed alternatives and chooses one rather than combining them. Its loss supports the usefulness of producing a new harness from the evidence pool. However, removing composition changes the sequence of operations, so the result should not automatically be described as a fully matched total-token comparison. The paper’s compute analysis and its structural ablations answer related but different questions.

The library experiment also has two distinct forms. Removing the library during the full workflow hurts performance. Separately, Figure 7 asks whether stored experience can support direct harness editing without obtaining new reference executions or candidate probes. On Code, direct editing without a library drops the score from the base 73.4 to 63.2. A library built with $k=1$ brings it to 73.8, and a library built with $k=5$ to 74.3. Web and Office also benefit from the richer library.

Figure 6. Redrawn from Figure 7. “No new probes” describes the application phase; the earlier library-building phase used execution evidence.

The contrast is striking because it shows that editing itself is not reliably beneficial. Without evidence or prior lessons, the editor can replace a workable process with a plausible but harmful one. At the same time, library-assisted direct editing does not recover the full probed system’s Code performance. Reuse economizes on new exploration while giving up some task-specific feedback.

Appendix E.1 removes components from the already refined $k=5$ harness without re-running optimization. Removing skills reduces the WorkBuddy mean from 80.8 to 75.5; removing scripts gives 77.1; removing the prompt-related group gives 79.3. This is evidence that the final harness depends on multiple component types.

Figure 7. Redrawn from Table 6. These are post-refinement removals, so the experiment does not test whether a fresh optimizer could compensate for a missing component type.

A removal loss is not a universal importance ranking. If the system had been forced to work without scripts throughout search, it might have found a different skill or tool strategy. Nor can the skill, script, and prompt losses be summed to recover the total improvement: their effects interact. The narrow, defensible conclusion is that the learned configuration uses all three groups productively under the reported intervention.

9. Transfer, reliability, and concrete mechanisms

9.1 Transfer has several meanings

Transferring a JobBench library to WorkBuddy yields Code, Web, and Office scores of 77.1, 76.9, and 82.4, compared with 76.5, 76.4, and 79.5 for the corresponding ordinary $k=1$ setup. This supports reuse across these professional-task domains. It does not establish that every library transfers or that unrelated task distributions benefit.

The paper also re-evaluates earlier tasks with a later library while excluding each task’s own entry. WorkBuddy rises from 77.5 to 79.4 and JobBench from 36.0 to 37.7. This is useful evidence about retrospective reuse. It is not a gain achieved by the original online stream at the time those earlier tasks arrived: later experience is available, and tasks are executed again.

Cross-model transfer and same-model editor–executor studies further weaken a purely “stronger teacher” explanation. Nevertheless, the main configuration still uses a strong improver, and executor, editor, library, and task distribution jointly determine performance. A transferable skill can supply a missing procedure to one model while adding redundant instructions to another.

Figure 10’s file-overlap analysis is informative but narrower still. It classifies modified files by overlap with library entries after excluding common boilerplate. A low textual overlap does not imply that an idea was invented independently, and high overlap does not establish causal value. The statistic describes material reuse; it is not a direct measurement of conceptual learning.

9.2 Fewer errors and fewer calls are different outcomes

Table 7 reports lower tool-error rates on all three benchmarks. WorkBuddy’s average tool calls fall from 20.0 to 17.8, while JobBench’s rise from 27.4 to 32.8 and LAB’s from 20.8 to 25.2. More reliable work can require more calls because the agent retrieves additional evidence or completes previously omitted checks.

Figure 8. Redrawn from Table 7 for Gemini 3.5 Flash/OpenCode at k=1. Call counts and error rates use different denominators and should be interpreted separately.

For example, the JobBench error rate changes from 14.4% to 9.1%. The absolute decrease is 5.3 percentage points, while the relative decrease is:

$$\frac{14.4-9.1}{14.4}\times100\%\approx36.8\%.$$

Neither number says that task correctness improved by 36.8%. A call may succeed while reading the wrong file, and a failed call can be recovered without harming the deliverable. The metric helps characterize execution behavior, not replace end-to-end grading.

9.3 The case studies explain why form matters

One coding case improves from 60 to 100 after tool guidance addresses pseudo-tool calls that were not being executed. The intervention targets the interface between text and action, rather than adding domain facts. An Office case improves from 23.6 to 100 after the harness corrects the aggregation grain and related checks. A Web case improves from 64 to 100 after changes address stale interface state and validation.

These examples are selected qualitative cases, not population-average effect sizes. Their value is to show how a concrete failure maps to an appropriate component. Repeated deterministic calculation is a candidate for a script; a recurring document procedure is a candidate for a triggered skill; a mechanical runtime condition can motivate a plugin. The paper’s eight-component case includes a plugin warning after 25 tool calls without an output artifact. A warning is not the same as a hard execution block.

The legal redlining case deserves a more careful reading. Its score rises from 44.4 to 94.5 when changes become visible to the evaluator’s text representation instead of remaining only in tracked-change metadata. That can improve evaluability, but the correct professional output format still depends on the user’s actual requirements. A benchmark’s extraction path must not silently become a universal rule for legal-document preparation.

10. Limitations: where the method can fail

The first limitation is cost. Probes, editors, composers, selection, and library maintenance consume resources beyond the final executor session. The fixed exploration budget can overspend on a straightforward task and still underspend on a difficult ambiguous one. Parallelism helps elapsed time only when resources and service limits permit it.

The second is evidence quality. Matching the environment prevents obvious contamination between trials, but a single probe does not separate an editing effect from run-to-run variation. Shared model priors also create correlated mistakes: several candidates can agree because they all inherit the same wrong premise. Increasing their number does not necessarily add independent evidence.

The third is selection error. A good candidate in the pool is not enough. The selector can prefer a superficially complete or concise answer whose omissions are harder to notice. The final selector is itself an agent, so the system has relocated part of the correctness problem rather than eliminated it.

The fourth is persistent contamination of the improvement library. A narrowly useful lesson can become overgeneralized, a task-specific fact can be misremembered as a stable rule, and an outdated procedure can be retrieved long after its assumptions change. The paper uses a simple library and acknowledges the need for consolidation and better retrieval as it grows.

Finally, benchmark scores and practical utility need not move together. Judge failures, inaccessible artifact representations, and all-or-nothing rubric items can exaggerate or hide operational changes. The appendix’s failure analysis is valuable precisely because it includes these complications instead of explaining every regression as a bad edit.

11. Critical analysis: what has really been demonstrated?

11.1 Separate candidate quality from selection quality

One Figure 14 Code case has candidate scores 95, 85, 85, 80, and 55. The selector chooses 55; the reference scores 96. These scores are known only retrospectively. During adaptation, the system cannot simply choose the numerical maximum.

Let $y_b$ be the best output in the candidate pool under the hidden evaluator and $y_s$ the selected output. Add and subtract $G(q,y_b)$ to decompose the final change:

$$\begin{aligned} G(q,y_s)-G(q,y_0) ={}&[G(q,y_b)-G(q,y_0)]\\ &-[G(q,y_b)-G(q,y_s)]. \end{aligned}$$

The first term is the pool’s potential improvement over the reference. The second is selection regret. For this example:

$$55-96=(95-96)-(95-55)=-1-40=-41.$$

This algebra is my explanatory decomposition, not a new theorem or an extra experiment. It shows why “search generated a good candidate” and “the system delivered a good answer” require separate measurements.

Figure 9. Redrawn from the WB-Code failure in Figure 14. The hidden scores explain the failure after evaluation; none is supplied to the selector.

A stronger experimental report would give pool-best score, selected score, and selection regret at each budget. If the pool improves with $k$ while regret also grows, the right next step may be a better selector rather than a larger search. An oracle-selection curve would be an analytical upper bound, clearly separated from deployable performance.

11.2 A self-generated check can reinforce the same mistake

Another failure assumes that a cache keyed by object identity is sufficient. The edited harness’s own checker reports success three times, while tests requiring content equivalence fail. This is not resolved by asking the same assumptions to approve themselves more often.

The key question is whether the probe can challenge the premise of the edit. A counterexample that changes object identity while preserving content would be more informative than another successful run on the original narrow case. This is a proposed diagnostic strategy, not an experiment reported here. It illustrates why validation should be designed around alternative explanations, not only around confirming that the intended action occurred.

The paper’s literal-header failure has a similar structure. A rule preserving exact key casing sounds conservative, but it caps the candidate pool when the task requires normalization. “Follow the format precisely” is not self-interpreting; the system must identify which constraints are actual requirements and which are assumptions introduced by the editor.

11.3 How much of the second recursion is better optimization?

Library-based improvement is a meaningful persistent change. However, the current evidence bundles several mechanisms: retrieving reusable files, recalling domain procedures, remembering failure patterns, and altering how editors diagnose problems. Calling the entire effect “learning to improve” is reasonable at the system level, but does not distinguish those mechanisms.

A useful follow-up would compare four conditions: no stored material, components only, lessons only, and both. Candidate counts, retrieval budgets, model choices, and total token budgets should be matched as far as practical. The experiment could then ask whether lessons improve proposals even when no executable template is available, and whether components remain useful without their explanatory context.

A second follow-up would freeze a library at one point in the stream and compare it with continued updates on later tasks. That would separate the benefit of possessing a library from the benefit of continually changing it. Testing task-order reversals and deliberate distribution shifts would help expose whether the library learns adaptable procedures or accumulates narrow shortcuts.

11.4 Interpret the computation claim at the right boundary

The paper’s observed compute–performance frontier is useful evidence within its measured methods and settings. It should not be generalized into a universal cost advantage. Offline tuning can be amortized across many deployments; online probes are paid again for each task; library construction and maintenance can be amortized differently. The preferred method depends on task volume, novelty, concurrency, and how costly an incorrect output is.

The practical question is not simply whether more recursion raises a mean score. It is whether the next unit of probing is worth its cost on this task. A proposed adaptive stopping policy would compare a calibrated expected quality gain with incremental cost, and stop when new candidates show no new failure mechanism or when available evidence cannot discriminate among them. The paper does not establish such a calibrated policy. Treating uncertainty as a reason to abstain from editing is at least as important as generating another proposal.

My overall assessment is positive but bounded. The method makes harness edits more evidence-driven, demonstrates gains beyond output-only scaling in the tested settings, and exposes instructive failures. Its strongest contribution is a concrete division between local experimentation and cross-task retention. Its weakest unresolved link is converting imperfect behavioral evidence into reliable adoption and durable lessons.

12. Conclusion

Harness² offers a useful way to think about agent self-improvement without changing model weights. On each task, propose scoped edits, observe their consequences, and compose a harness that is executed before adoption. Across tasks, retain reusable components and supported experience while resetting the next execution to a common base.

The results support this system design across multiple professional benchmarks and model–harness combinations. They also show why the attractive phrase “self-improvement” needs careful accounting. More candidates can improve the pool while a selector still chooses badly; a checker can confirm its own mistaken premise; a reusable procedure can carry an outdated assumption; fewer executor steps can coexist with much more total computation.

The takeaway I would keep is simple: an agent improves more credibly when its edits are tied to observable changes, and those observations remain distinguishable from claims of correctness. Harness² advances that discipline. It leaves selection reliability, causal attribution, library maintenance, and adaptive compute allocation as substantial research problems.

References and figure provenance

1. Xu, R., Chen, Y., CuiZhu, Z., Mishra, B. D., Ming, Y., Yu, H., Han, R., Lee, C.-Y., and Pfister, T. Harness^2: Recursive Agent Harnessing for an Open World. arXiv:2610.14224v1, 2026-10-09. Method: Sections 2.1–2.3; protocol: Section 3 and Appendix D; results: Tables 1–3; additional analyses and failures: Appendix E; limitations: Appendix F; full stage prompts: Appendix H.
2. Author project page and official resources, linked as resource entry points.

Figures 1–3 are original explanatory diagrams based on the paper. Figures 4–9 redraw reported numerical data or failure examples with source locations in their captions. The source paper is distributed under CC BY 4.0; this review attributes the original authors and distinguishes explanatory calculations from their measurements. No figure represents a new benchmark run.