AutoHarness: Learning the Rules Around an LLM Agent

Review date: October 11, 2026
Author: Zhongzhu Zhou
Paper reviewed: AutoHarness: improving LLM agents by automatically synthesizing a code harness
Paper authors: Xinghua Lou, Miguel Lázaro-Gredilla, Antoine Dedieu, Carter Wendelken, Wolfgang Lehrach, Kevin P. Murphy
Paper / DOI: arXiv:2603.03329v1 / 10.48550/arXiv.2603.03329
Version and dates: v1; arXiv lists February 10, 2026; the PDF title page is dated March 5, 2026.

1. The contribution: improve the program around the model

AutoHarness asks a practical question: if a language model repeatedly violates the rules of an environment, can the same model write a reusable program that stops those mistakes? Its answer is to search over small Python harnesses using executable feedback. The model proposes edits, the environment supplies evidence of failure, and a search tree preserves alternative programs. This is optimization of an agent’s surrounding program, not an update to the language model’s weights. The distinction matters when discussing agents that improve themselves.

The paper develops three operating points. An action filter proposes candidate moves that an LLM can rank. An action verifier checks an LLM proposal and requests another when it fails. A complete code policy chooses the move without invoking an LLM during game play. Most experiments concern the verifier; the strongest aggregate single-player reward comes from the complete code policy. These are different systems with different cost profiles, rather than interchangeable names for one method.

Three evidence sets deserve separate treatment. Appendix Table 1 reports perfect observed legality for learned harnesses across 145 TextArena environments. End-to-end agent comparisons use only 32 games: 16 single-player and 16 two-player. Finally, code-policy reward comparisons use the 16 single-player games. Success on the first set is not equivalent to strategic dominance across all 145 environments. Moreover, Appendix Figure 7 reports residual illegal actions for several LLM-plus-harness agents, despite the perfect standalone results. This review returns to that distinction rather than smoothing it away.

The most useful takeaway is that narrow, repeatable obligations can be moved from token generation into a program. A model need not rediscover the rules of movement at every turn. But the reliability of the resulting system depends on the correctness of the learned predicate, the completeness of the observed state, and what the control loop does when it cannot obtain an accepted action. All three sit outside a headline average reward.

Figure 1 (paper Fig. 1): The search tree contains alternative harness programs. Execution supplies evidence for a critic and refiner; a child is a new program, not a new game state. Cropped from Lou et al., AutoHarness v1, CC BY 4.0.

2. Prerequisites: legality, observation, and reward

Let the environment have state $s$, with a text observation $o=g(s)$. An action is a string $a$ interpreted by the environment. Define the true applicability predicate $L(s,a)$ to be one exactly when the action is permitted. The model’s proposal distribution is $q_\theta(a\mid o)$. A synthesized verifier $v(o,a)$ is a program that returns either zero or one. These quantities live at different levels: the environment knows the rule, the model proposes a string, and the verifier predicts whether that string satisfies the rule.

A legal action can still be strategically poor. In chess, moving a piece to a legal square may immediately lose a queen. In Minesweeper, selecting a previously hidden cell can be a valid action even if the cell contains a mine. Conversely, a plausible strategic intention can be rejected because its string encoding is malformed. Format validity, rule validity, and expected reward should therefore be measured separately. A legality score alone does not explain which of these problems was solved.

There is also an information constraint. If two hidden states share the same observation but disagree about the legality of some action, an observation-only verifier cannot classify that action correctly in both states. Formally, an exact such verifier requires the following condition on states that can occur:

$$g(s_1)=g(s_2)\quad\Longrightarrow\quad L(s_1,a)=L(s_2,a)\quad\text{for every relevant }a. \tag{1}$$

This is a necessary information condition, not a theorem proved in AutoHarness. It explains why state tracking matters. A chess board diagram alone may omit castling rights or en-passant history. The full environment observation might include the missing information, but a reader should check that contract rather than assume a board parser has recovered the complete game state. More model capability cannot reconstruct information that the input never provides.

Finally, the word training here means iterative program synthesis with rollouts. The language model remains the generator of candidate edits. The environment evaluates behavior, while the tree search decides where to spend the next synthesis call. This makes AutoHarness closer to feedback-driven program search than to gradient-based reinforcement learning of the LLM.

3. What rejection actually conditions on

The paper describes a verifier harness as a learned rejection sampler. A simple derivation makes the promise and its boundary explicit. Hold an observation fixed, draw independent proposals from $q_\theta$, and accept only actions for which $v(o,a)=1$. The accepted probability mass is

$$Z(o)=\sum_a q_\theta(a\mid o)v(o,a). \tag{2}$$

The event of accepting action $a$ has probability $q_\theta(a\mid o)v(o,a)$. Dividing by the probability of accepting anything gives the conditional policy:

$$q_v(a\mid o)=\frac{q_\theta(a\mid o)v(o,a)}{Z(o)},\qquad Z(o)>0. \tag{3}$$

An exact verifier would condition on genuine legality. A learned verifier conditions on its own acceptance set. A false positive therefore remains reachable, while a false negative deletes a potentially valuable legal action. This is why a high acceptance rate and a low illegal-action rate are both necessary but still insufficient for strong decision making.

Under the same independent, unchanged-proposal assumption, the probability that the first accepted proposal occurs on call $k$ is $(1-Z)^{k-1}Z$. Summing the tail probabilities of the geometric distribution gives

$$\mathbb{E}[N]=\sum_{k=0}^{\infty}(1-Z)^k=\frac{1}{Z}. \tag{4}$$

Thus a verifier can improve action quality while multiplying model calls. If only one in ten proposals is accepted, the idealized expected number of proposals is ten. A system that reports only the final valid action hides this expense. The paper’s real loop appends a warning after rejection, changing the prompt and hence the proposal distribution. Equations (3) and (4) are explanatory baselines, not an exact description of that adaptive loop or measured latency predictions.

For a fixed proposal distribution and at most $K$ attempts, failure to find any accepted action has probability $(1-Z)^K$. Equivalently,

$$P(\text{accepted by }K)=1-(1-Z)^K. \tag{5}$$

A finite budget turns verifier quality into a systems question: what happens after the last rejection? Abstention, a deterministic legal fallback, and an unchecked final proposal have different effects on reward and legality. AutoHarness’s broad conceptual description should not be read as specifying all such runtime behavior. The distinction is particularly relevant to the appendix’s residual errors.

Figure 2: Original mathematical illustration of Equations (4) and (5), using independent proposals and a fixed verifier. The curves are analytical, not measurements from AutoHarness. Low acceptance can make a seemingly cheap guard expensive.

4. A verifier’s two different errors

Let $p$ be the original probability of a legal proposal at a particular state. Let $\alpha$ be the probability of rejecting a legal proposal, and $\beta$ the probability of accepting an illegal proposal. These are defined relative to the proposal distribution, so they may change when the model or prompt changes. Partitioning proposals into legal and illegal events gives

$$Z=p(1-\alpha)+(1-p)\beta. \tag{6}$$

The illegal portion of the accepted mass is $(1-p)\beta$. Consequently the illegality rate among accepted proposals is

$$P(\text{illegal}\mid\text{accepted}) =\frac{(1-p)\beta}{p(1-\alpha)+(1-p)\beta}. \tag{7}$$

Consider an illustrative case with $p=0.8$, $\alpha=0.1$, and $\beta=0.05$. The accepted mass is $0.72+0.01=0.73$, and the residual illegal rate is about $1.37\%$. This is a calculation, not a reported AutoHarness result. It shows why a checker that rejects most bad moves can still be unreliable over a long game. Reducing false positives matters more than merely displaying a large improvement relative to the original model.

False negatives have another cost. Suppose a puzzle has exactly one winning move, and a learned checker rejects it. The remaining accepted moves can all be legal while guaranteeing failure. A legality-only evaluator may never punish this error if its rollouts use a proposal function that avoids the rejected move. The action proposer and verifier can therefore agree with each other while jointly omitting important parts of the legal action space.

The action-filter variant moves the problem upstream: code produces a candidate set and the LLM ranks it. This can avoid repeated rejection, but the set must include good moves. A perfect ranker cannot recover a move that the generator omitted. The verifier variant retains the LLM’s broad proposal ability but pays for rejection and risks false acceptance. The complete code policy removes inference-time model calls but must encode enough strategy itself. These design choices trade candidate coverage, runtime cost, and strategic expressiveness.

Algorithm 1. Verifier control loop, with an explicit boundary

The central propose-check-retry behavior follows Section 3. The finite budget and abstention below make an engineering boundary explicit; they are not claimed as the paper’s exact runtime policy.

1. Receive observation o, frozen verifier v, and attempt budget K.
2. Form the model prompt from the rules and current observation.
3. For attempt = 1, ..., K:
4.     Ask the LLM for one candidate action a.
5.     If v(o, a) accepts: return a to the environment.
6.     Otherwise append a rejection warning to the next prompt.
7. If all attempts fail: abstain or invoke a separately specified fallback.
8. Log rejections, fallback use, and the environment's final verdict.

5. Search over programs, not over next moves

AutoHarness keeps multiple program hypotheses in a tree. A parent is a previous harness; its child is a refinement produced by the LLM. The critic consolidates errors from environment execution, and the refiner receives those errors together with the previous program. Retaining branches matters because a repair can improve one class of states while breaking another. Greedily overwriting the single current program would discard alternatives that might later become useful.

The paper says node selection uses Thompson sampling, following Tang et al., and sets the heuristic weight to 1.0. Its main text does not give a full posterior family, update rule, or numerical mapping from heuristic values to sampling probabilities. It would be inaccurate to manufacture those details and label them AutoHarness’s exact algorithm. Conceptually, Thompson sampling draws a plausible quality for each choice from an uncertainty model, then acts greedily under that draw:

$$\widetilde{u}_i\sim P(u_i\mid D_i),\qquad i^*=\operatorname*{arg\,max}_{i}\widetilde{u}_i. \tag{8}$$

Here $D_i$ is evidence associated with candidate $i$. Equation (8) explains the exploration principle; it is not a recovered implementation specification. A candidate with uncertain quality can occasionally beat a well-tested incumbent under a sampled draw. That allows exploration without choosing entirely at random. In program search, however, the quality of a parent and the promise of its future children are different concepts. A mediocre parent might be one small repair away from a strong child. The paper does not isolate which uncertainty model best captures this distinction.

For verifier synthesis, each iteration uses ten parallel environments, with rollouts up to 1,000 steps and automatic resets. A rollout stops when an illegal action or program failure occurs. Up to five failed steps are sampled for the critic. Training stops when the heuristic reaches 1.0 or the time budget expires. Gemini-2.5-Flash supplies the code refinements. A small feedback bundle saves context and highlights failures, but may miss a rare error class; repeated independent testing is therefore valuable.

The method description distinguishes two failure paths. If the checker accepts an action that the environment rejects, both the proposer and verifier can be refined. If the checker rejects an action that is indeed invalid, the proposer is refined. The external environment supplies the authority on whether the attempted action was legal. The checker should not become its own final judge, because agreement between two generated functions is weaker evidence than agreement with the environment.

Algorithm 2. Harness synthesis at the paper’s stated level

This numbered reconstruction includes the documented control flow. The node sampler remains abstract where the paper leaves its distribution unspecified.

1. Initialize a tree with a candidate harness and its evaluation evidence.
2. Select a program node with Thompson-sampling-guided search.
3. Roll out its behavior in 10 parallel environments, up to 1,000 steps.
4. Stop a rollout on an illegal action or an execution failure.
5. Sample at most 5 failed steps; consolidate them into critic feedback.
6. Ask Gemini-2.5-Flash to refine the program using that feedback.
7. Evaluate the new program and add it as a child hypothesis.
8. Update search evidence using the legal-action heuristic.
9. Stop at the target heuristic or budget limit; otherwise repeat.
10. Assess the selected harness on fresh test rollouts.

Figure 3 (paper Fig. 2): Six selected learning curves. Long plateaus and late jumps illustrate why program refinement can require multiple attempts. The figure does not establish that Thompson sampling is better than other search rules. Cropped from AutoHarness v1, CC BY 4.0.

6. Turning a checker into a complete policy

The policy variant changes the goal of the action proposer. Instead of proposing a valid random move, it should propose a good legal move that maximizes the final reward. The generated program uses ordinary Python functionality and libraries such as NumPy, with no LLM invocation at decision time in the reported setting. This is a substantive change: reasoning expense moves from repeated model calls into one-time synthesis and subsequent program execution.

The policy search must reward strategy as well as legality. Section 4.3 defines the trajectory heuristic using terminal reward $r\in[0,1]$:

$$H(\tau)= \begin{cases} 0, & \text{if trajectory }\tau\text{ contains an illegal action},\\ 0.5+0.5r(\tau), & \text{otherwise}. \end{cases} \tag{9}$$

The lower half of the range is unused except for zero. A completely legal but unsuccessful trajectory scores 0.5, while a successful legal one can reach 1.0. Thus a single illegal action can erase all strategic progress on that rollout. This encodes a strong preference for applicability, but averages of this heuristic do not create an absolute constraint on every trajectory.

To see that boundary, write $\ell=P(\tau\text{ is legal})$ and let $\bar r_L$ be the mean reward conditional on a legal trajectory. Taking expectations gives

$$\mathbb{E}[H]=0.5\ell+0.5\ell\bar r_L =\frac{\ell(1+\bar r_L)}{2}. \tag{10}$$

A hypothetical policy that is always legal with reward 0.2 scores 0.6. Another that is legal only 80% of the time but obtains reward 1 on those legal trajectories scores 0.8. Hence optimizing mean $H$ can prefer the less reliable policy. This does not invalidate the reported policy results; it clarifies what the objective alone can guarantee. A hard reliability requirement would need a separate acceptance condition or constrained objective.

Figure 4: Original plot of the paper’s policy heuristic, Equation (9). Legal trajectories receive a reward-dependent score; illegal trajectories receive zero. The plot represents the objective, not empirical training observations.

The authors allow up to 256 synthesis iterations for code policies, reporting a mean of 89.4 iterations and a mean heuristic of 0.939. These numbers cannot be compared directly to 14.5 verifier iterations as though the workloads were identical: the policy has a harder objective, and its evaluation set is the 16 single-player games. The paper does not show complete-code policies for the two-player suite, where reasoning about an opponent can require much more sophisticated planning.

Algorithm 3. Reward-aware policy search

This reconstruction emphasizes the change in evaluation objective while reusing the same program-search idea.

1. Give the refiner the game's rules, observation format, and action interface.
2. Generate a candidate program that directly proposes a good legal action.
3. Execute full trajectories and collect legality and terminal reward.
4. Assign H = 0 to any illegal trajectory; otherwise H = 0.5 + 0.5r.
5. Use the search tree and execution feedback to propose another program.
6. Continue within the synthesis budget, at most 256 iterations here.
7. Freeze the selected program and test it on fresh game instances.
8. During test play, call the program without calling an LLM.

7. What the experimental protocol actually covers

The source pool contains one-player and two-player TextArena games. The authors remove nine games with free-form text or dialogue actions, leaving 145 environment variants. Some observations are manually modified to remove lists of available moves. Appendix A illustrates this for chess: the board and action format remain, while the explicit list of valid moves disappears. This removes an easy shortcut and makes the applicability task more meaningful.

It also narrows the generalization claim. A finite action grammar with environment-supplied legality feedback differs substantially from a research assistant judged on an open-ended report. The excluded dialogue games are precisely settings where an action may be grammatical yet pragmatically poor, with no clean Boolean oracle. AutoHarness demonstrates an effective mechanism where the environment can expose violations. It does not establish that arbitrary assistant quality can be reduced to the same predicate.

The standalone harness test uses fresh rollouts described as length 1,000 across ten random seeds per game. All entries in Table 1 have legal-action rate 1.0. The end-to-end reward evaluation is smaller for efficiency. Single-player agents run 20 matches per game, while the two-player comparison uses 40 matches with first and second player roles split evenly. The same optimized game-playing prompt is used across the compared models. This controls one source of variation but does not equalize total inference tokens, retries, or synthesis expense.

For the complete-policy comparison, GPT-5.2 and GPT-5.2-High receive only ten and five repetitions per game respectively, due to cost. Other methods receive twenty. Unequal counts do not automatically bias a sample mean, but they change uncertainty. A gap of a few hundredths between mean rewards should not be treated as a confident ranking without interval estimates or a paired design. Five runs also provide little evidence about rare failures.

The 145 environments include many variants of shared game families. This is useful for testing multiple configurations, but counting every variant as an independent new task overstates semantic diversity. Table 1, for example, contains many PigDice variants requiring one learning step. A family-level summary would complement the environment-level mean and reveal whether success depends on a few recognizable templates.

The protocol therefore supplies several useful but non-equivalent observations: broad sampled legality, narrower actual game strength, and a preliminary code-policy result. Readers can value all three without merging their denominators. For applications, the matching question is whether one’s deployment resembles rule-governed game states, strategic competition, or open-ended tasks outside the evaluated interface.

8. Aggregate reward: a meaningful gain with a small margin

For the 16 single-player games, the paper reports mean rewards of 0.673 for Gemini-2.5-Flash, 0.707 for Gemini-2.5-Pro, and 0.745 for Flash with a harness. The harness beats Pro on eight games and ties on five. This is a concrete demonstration that modifying the surrounding program can be more effective than switching to the larger model for this benchmark selection.

The absolute gain over unassisted Flash is about 0.072 reward, and over Pro about 0.038. These are reward-point differences, not percentage-point changes in a universal task success rate. The games have different reward semantics, and their mean gives equal weight to each game. Relative improvement can be calculated, but the denominator must be explicit and should not be confused with a general capability improvement.

Figure 5: Redrawn from the displayed values in paper Fig. 6, with each of the 16 games weighted equally. Means recomputed from the table agree with the reported three-decimal aggregates. These are paper results, not new experiments.

The complete code policy reaches 0.870 mean reward, compared with 0.844 for GPT-5.2-High. The 0.026 advantage is interesting because code policies avoid test-time model calls. It is also small enough that uncertainty matters. The report does not provide a significance test establishing that the ordering would persist under many independent reruns. The appropriate conclusion is that the synthesized policy is highly competitive on the evaluated aggregate, with an especially attractive inference structure.

Recomputing means from the three-decimal cells in Appendix Figure 6 yields 0.6725, 0.7074375, 0.74475, 0.635125, 0.844, and 0.869875 in the order shown. These simple arithmetic checks support the main comparisons. They also establish the weighting: every game contributes one sixteenth, regardless of the number of matches played for a particular model. Recomputing an average verifies arithmetic, but it cannot recover the variation between runs. The latter requires raw outcomes, which are not provided in this table.

9. Strategy is heterogeneous across games

The two-player experiment compares Flash plus its learned harness with Pro, reporting an overall win rate of 56.3% for the harness agent and 38.2% for Pro, with draws accounting for the rest. The harness agent wins the game-level comparison in nine of sixteen games. Against vanilla Flash it wins twelve of sixteen game-level comparisons, with a reported overall win rate of 64.8%. Game-level wins and match-level win rate are different statistics; nine divided by sixteen happens to be close to 56.3%, but that numerical coincidence does not make the measures identical.

Figure 6 (paper Fig. 3): Match outcomes against Gemini-2.5-Pro for the 16 two-player games. The large variation across bars limits any claim of universal strategic superiority. Cropped from AutoHarness v1, CC BY 4.0.

The complete code policy’s single-player aggregate also hides a striking tradeoff. Against GPT-5.2-High it wins only three games, loses five, and ties eight. Large improvements on a few games outweigh smaller losses on more games. In the displayed table, code policy gains 0.571 on PegJump, 0.414 on FifteenPuzzle, and 0.167 on 2048. It loses 0.541 on Bandit, 0.070 on Blackjack, 0.060 on Minesweeper, 0.050 on Secretary, and 0.017 on Sokoban.

Summing those displayed differences gives 0.414 across all sixteen games, or 0.025875 per game. This reproduces the aggregate advantage without implying that code is preferable for a randomly chosen application. A deployment dominated by bandit-like exploration would care much more about the negative Bandit result than about the average. A workload of deterministic puzzles might value the positive cases differently.

Figure 7: Redrawn from paper Fig. 6. Shown are the eight non-tied games between GPT-5.2-High and the synthesized policy; the other eight tie. Reward differences explain why the higher mean can coexist with fewer game-level wins.

There is a plausible mechanistic interpretation, but it should remain an interpretation. Some puzzles admit compact deterministic procedures that are efficient to execute once discovered. Other environments require exploration, uncertain inference, or adaptation that a synthesized heuristic may handle poorly. The paper does not conduct a causal study assigning each reward gain to a specific algorithmic ingredient, so one should not infer from a game name that a particular solver was necessarily learned.

The paper’s appendix illustrates deduction-oriented Minesweeper logic and local chess-rule reasoning. Those examples help explain the style of program that can be synthesized, but excerpts are not comprehensive evidence about all tested policies. The key experimental question is the behavior of the frozen agent on fresh instances, including cases where its heuristic is wrong. This review relies on the paper and its appendix, and does not claim to have reproduced the game results.

10. The legality gap that the headline does not resolve

Table 1 gives every learned harness a legal-action rate of 1.0. Appendix Figure 7, however, shows that Flash plus the harness has 99.86% on 2048, 99.77% on Bandit, 96.59% on FifteenPuzzle, 99.76% on LightsOut, 98.25% on PegJump, 97.24% on RushHour, and 98.48% on Sokoban. The other nine displayed single-player entries are 100%. The complete code-policy column is 100% on all sixteen. These are directly visible source values, not estimates inferred from bar heights.

Figure 8: Redrawn from paper Fig. 7 by subtracting the Flash-plus-harness legal rate from 100%. Only the seven games with nonzero residual error are shown. This is a different evaluation from Table 1’s standalone harness rollouts.

The two tables need not be logically inconsistent if they exercise different distributions or control paths. A harness tested with its own proposal generator may see a narrow set of plausible actions. An LLM may produce malformed strings, unusual legal moves, or boundary cases outside that set. Strategic game play can also visit states not reached by random legal proposals. Such distribution shifts are plausible explanations, but the paper does not provide the detailed error trace analysis needed to identify the cause.

Likewise, retry exhaustion, parsing, and fallback behavior could affect end-to-end results, but none should be asserted as the observed root cause without evidence. The defensible claim is narrower: perfect legality in the reported standalone protocol did not translate into perfect legality for every end-to-end LLM-plus-harness result. This is an important measurement distinction, not a reason to discard the overall improvement.

For an action checker, false-positive testing should deliberately include invalid actions, not just observe that its own proposer produces valid ones. False-negative testing should include a diverse set of known legal actions, especially actions a strategic model may prefer. The environment’s accept/reject verdict can provide labels for both. A test set built only from accepted behavior underestimates the region where the checker is most likely to fail.

There is also a denominator issue. A move-weighted accuracy emphasizes long games with many easy moves; an episode failure rate answers whether a user can finish a task without an illegal action. Both are useful. If errors were independent with per-move probability $e$, a trajectory of $T$ moves would survive without error with probability $(1-e)^T$. Real errors are correlated and depend on state, but the elementary calculation shows why a percentage near 100% may still be inadequate for long-horizon use.

11. Synthesis effort and the economics of reuse

Section 4.1 reports a mean of 14.5 refinement iterations and says nineteen of thirty-two games finish in fewer than ten. Recomputing the starred subset of Table 1 reproduces both statements exactly. Across all 145 listed variants, the arithmetic mean is about 10.30 steps. Thus the 14.5 mean describes the 32 selected games, not all 145 variants. Making the denominator explicit resolves an ambiguity in the prose.

The distribution is uneven. Chess-v0 needs 64 steps, Othello-v0 62, Cryptarithm-v0 45, and GermanWhist-v0 43. Across the full table, Breakthrough-v0-small needs 136 steps. Many variants need only one. A median or mean alone would conceal this tail, which matters if a product must provision a synthesis budget before seeing a new environment.

Figure 9: Recomputed from all 145 rows of paper Table 1 and its 32 starred rows. The histogram compares overlapping sets, not independent samples. The selected-set mean is 14.5; the full-table mean is 10.30.

Code policies make reuse especially attractive because deployment no longer requires an LLM call per decision. Let $C_s$ be one-time synthesis cost, $c_b$ the baseline cost per episode, and $c_h$ the harness system’s cost per episode. For $M$ episodes, the harness is cheaper when

$$C_s+M c_h<M c_b \quad\Longleftrightarrow\quad M>\frac{C_s}{c_b-c_h},\qquad c_b>c_h. \tag{11}$$

The break-even point depends on reuse, not just low runtime cost. Synthesis for a game played once may be wasteful; the same program serving thousands of instances may be attractive. Environment updates create maintenance costs and can shorten the usable lifetime. For the verifier variant, retries may even make $c_h$ exceed $c_b$, in which case pure cost savings never arrive under this simple model.

The paper reports approximately USD 640 for the GPT-5.2 and GPT-5.2-High experiments, and describes code-policy test-time cost as nearly zero. That supports the structural advantage of avoiding model calls, but it is not a complete amortized cost comparison. The source does not give all costs needed to evaluate Equation (11), including generation tokens, failed branches, rollout execution, and maintenance. Ordinary program execution is also not literally free. A useful deployment study would report both total dollars and wall-clock latency under matched workloads.

12. What zero observed errors can establish

Suppose a fixed verifier is evaluated on $n$ independent trials drawn from the intended deployment distribution. If its true error probability is $e$, the probability of observing zero errors is $(1-e)^n$. Inverting a one-sided 95% test yields an upper error bound:

$$(1-e_{\mathrm{upper}})^n=0.05 \quad\Longrightarrow\quad e_{\mathrm{upper}}=1-0.05^{1/n} \approx\frac{-\log(0.05)}{n}\approx\frac{3}{n}. \tag{12}$$

The approximation follows by writing $0.05^{1/n}=\exp(\log(0.05)/n)$ and using $\exp(x)\approx1+x$ for small $x$. With 1,000 genuinely independent error-free trials, the upper bound is about 0.003, not zero. With 10,000 it is about 0.0003. These numbers are illustrations of sampling uncertainty; they are not confidence intervals calculated from AutoHarness’s raw trajectories.

Figure 10: Original analytical plot of Equation (12). The independent-trial assumption is essential. Consecutive game states, adaptive model selection, and family-related tasks can make the nominal trial count a poor measure of independent evidence.

Game trajectories complicate the calculation. Neighboring moves share a board and history, so failures can cluster around the same special rule. Different variants may reuse the same latent knowledge. Training stops after reaching perfect observed accuracy, which selects an unusually successful candidate from many attempts. Reusing those same evaluations as final evidence would introduce selection bias. The paper does use fresh test rollouts, which is valuable, but it does not provide enough granular counts and dependencies for the simple bound to become a formal guarantee.

The right evaluation unit depends on the intended claim. To estimate whether an episode finishes without violation, independently seeded episodes are more natural than individual moves. To estimate transfer to a new game family, independent families matter more than many variants of a familiar game. To claim correctness for all states, statistical testing is insufficient: the action semantics and program need stronger reasoning than sampled rollouts.

This distinction also tempers strategic comparisons. The reported 0.026 reward margin over GPT-5.2-High is a mean of heterogeneous game outcomes with unequal repetition counts. A paired bootstrap over games addresses variability across games; repeated matched seeds address variability within games. Neither can be reconstructed faithfully from only the displayed means. Presenting exact confidence intervals here would fabricate information the paper does not supply.

13. Limitations of the demonstrated result

First, each environment receives a separate harness. This provides specialization but leaves open how well rules transfer between games, versions, or observation formats. The conclusion proposes reusable libraries and distilling experts back into the base model as future work. The present experiments do not demonstrate recursive improvement of the model itself. The system improves an artifact produced by the model.

Second, the evaluation omits free-form text-action games. That choice keeps applicability measurable but excludes much of what makes general assistants difficult. An action can be syntactically allowed yet violate an unstated preference, use a tool at the wrong time, or make an unsupported scientific assertion. Extending the method requires specifying a reliable feedback signal, not merely putting a code generator around an assistant.

Third, the method’s most distinctive search decision lacks a controlled ablation in the reported experiments. We see that the whole procedure works, but not how much comes from tree retention, Thompson sampling, critic consolidation, multiple seeds, or simply repeated code generation. Random restart and linear repair are important alternatives because they can be simpler to run and easier to interpret.

Fourth, raw error counts, timing distributions, complete per-method budgets, and uncertainty estimates are incomplete. These gaps do not erase the reward tables, but they limit claims about universal reliability, statistically robust model rankings, and total cost. A useful technical reading should distinguish evidence absent from evidence negative: an unreported comparison is unresolved, not automatically unfavorable.

Finally, generated programs inherit the interface assumptions supplied to them. A change in board formatting, action grammar, timeout behavior, or hidden-state representation can make a once-successful harness unsuitable. The feedback loop may repair such changes when given a fresh budget, but that repair burden is part of the system’s lifecycle. A frozen checker is not automatically a durable specification of an evolving environment.

14. Critical analysis: what should the next study resolve?

Separate a good proposer from a good checker. The sharpest unresolved issue is whether the legality test measures broad discrimination or only safe behavior on a restricted action distribution. A proposer can avoid every difficult case, allowing a weak checker to look perfect. The next study should evaluate the checker on three pools: proposals from its own code, proposals from several frozen LLMs, and deliberately generated boundary actions with environment labels. The third pool should cover malformed syntax, out-of-range indices, rare rules, and known legal strategic moves. Report false acceptance and false rejection separately. This directly tests the mechanism behind the difference between Table 1 and Figure 7.

Measure opportunity cost from rejecting valid actions. The current headline emphasizes illegal actions, yet checker conservatism can erase good strategies. An oracle that always rejects everything has no false acceptances, but cannot play. Acceptance coverage and retry exhaustion should therefore appear beside legality. On small games where the complete legal set is available to the evaluator, measure the fraction retained and whether high-value legal actions are retained. This evaluator can use privileged labels without giving them to the acting agent. That preserves the harder observation setting while revealing over-restriction.

Make the search comparison budget-equivalent. A compelling ablation would compare independent samples, a single repair chain, tree search with uniform node choice, and Thompson-guided tree search. Match synthesis tokens, environment steps, and wall-clock limits; report best held-out reward as a function of each budget. Counting only iterations is insufficient because one candidate can consume far more feedback and reasoning tokens than another. Multiple synthesis seeds would reveal whether a good result depends on a lucky early program. This study would show when the search machinery earns its complexity.

Separate adaptation from generalization. Synthesizing one program for each game demonstrates adaptation using feedback, not immediate transfer to unseen game rules. A stronger test would hold out a game family or change a rule after synthesis, then plot performance against the new feedback budget. Another would change only observation formatting while preserving semantics. The former tests conceptual adaptation; the latter tests interface robustness. Combining them in one number makes it impossible to tell whether a failure comes from strategic knowledge or parsing fragility.

Use reliability constraints when reliability is the objective. Equation (10) shows that expected heuristic value can trade reliability against reward. If violating a rule is unacceptable, select candidates only after they pass a predeclared legality criterion, then rank the survivors by reward. The criterion should use held-out episodes and include a retry-exhaustion policy. A constrained design can still fail under distribution shift, but its selection rule would align more directly with the stated goal than a single blended score.

Report an amortization curve, not one cheap endpoint. For code policies, the natural comparison is cumulative cost over repeated tasks. Plot synthesis plus deployment expense against the number of served episodes, include repair after environment changes, and use the same reward target when comparing systems. An inexpensive policy that fails a crucial workload is not made useful by a low average price. Conversely, a costly synthesis process may be entirely justified by long-lived reuse. This framing makes the paper’s systems contribution testable without assuming that inference-time model calls are the only cost.

These proposals are evaluation designs, not experiments performed for this review. They preserve the paper’s appealing idea while asking it to meet clearer standards. The key opportunity is to treat generated harnesses as independently testable decision components, each with an explicit input contract and measurable failure distribution.

15. A reading and evaluation checklist

To interpret or extend the paper, begin by freezing the evaluation definitions. Record which environment variants have their valid-action hints removed, which seed pools are used for synthesis and final evaluation, what counts as an illegal move, and whether exceptions and exhausted retries are included in failure rates. State whether the legality denominator is proposed moves, accepted moves, submitted moves, or completed episodes. Those denominators can produce very different-looking percentages from the same execution history.

Next, keep an evidence trail for each synthesized candidate: its search parent, synthesis budget, observed failure categories, and held-out score. This is an experimental reporting recommendation. It does not require claiming that every generated function has been formally verified. The point is to distinguish selection evidence from test evidence, and to make regressions visible when a local repair affects other states.

For reward comparisons, use matched seeds where practical and preserve the first-player balance in two-player games. Report per-game outcomes alongside equal-game and workload-weighted averages. If a strong baseline gets fewer runs because it is expensive, state that constraint and quantify the resulting uncertainty when raw outcomes permit it. A higher mean with wider uncertainty can still be useful evidence, but it should not be summarized as a definitive ordering.

For cost comparisons, distinguish the one-time search bill from per-episode model calls and program execution. Log model calls that were rejected, not just the final selected action. Record both average and tail latency because an agent waiting through many retries experiences a different service from one with a direct code policy. It is often the slow, hard state rather than the average easy state that determines usability.

Finally, a future transfer study should distinguish three types of novelty: a new seed in the same game, a new configuration in the same family, and a genuinely new rule system. AutoHarness’s present evidence is strongest for specialized harnesses evaluated on fresh rollouts of the target environments. Naming that scope makes the result easier to reuse and harder to overstate.

16. Conclusion

AutoHarness offers a useful model of agent improvement: use the LLM to build a program once, test it through interaction, and let that program carry repeatable obligations at runtime. The verifier improves a smaller model’s game performance; the complete policy removes model calls during inference and achieves a strong single-player aggregate. Both results support investing in the program around the model.

The most important caveat is also actionable. A checker that looks perfect on one rollout distribution is not necessarily perfect inside an LLM-driven agent. The appendix makes that distinction visible, and the next step is to measure it directly. Stronger legality tests, budget-matched search ablations, and explicit amortization would turn a promising synthesis mechanism into a much clearer reliability and efficiency story.

References and figure provenance

1. Xinghua Lou et al. AutoHarness: improving LLM agents by automatically synthesizing a code harness, v1. This review reads all 21 PDF pages, including Appendices A-D. Methods and experiment claims above refer to Sections 3-4; detailed values refer to Table 1 and Figures 6-7.
2. AutoHarness full paper. Figure crops retain their source numbering; redrawn charts identify their source data. The paper’s CC BY 4.0 license permits attributed reuse. Original analytical curves are marked as such.
3. Hao Tang et al., Code repair with LLMs gives an exploration-exploitation tradeoff, cited by AutoHarness as its search antecedent. The present review does not substitute that prior work’s uninspected details for missing AutoHarness specifications.

All reward and legality measurements are attributed to the paper. Arithmetic recomputations and explanatory derivations are labeled; no new game experiments are claimed. The arXiv date and PDF date are retained separately because the source presents both.