ADAS: What Changes When Agents Design Other Agents?

Review date: 2026-10-17
Paper: Automated Design of Agentic Systems
Authors: Shengran Hu, Cong Lu, Jeff Clune
Source: arXiv:2408.08435v2, revised 2025-03-02; first submitted 2024-08-15
Resources: Author project page · Official code resource

1. The question behind the paper

An agent can fail even when its language model contains the knowledge needed to answer. It might ask the wrong intermediate question, discard useful feedback, let several copies repeat the same mistake, or combine good partial answers badly. Improving the surrounding program can therefore be valuable without updating the underlying model. The difficult part is deciding which surrounding program to use.

I read ADAS as an attempt to turn that design problem into an experimental search process. A designer model proposes an agent as a program, the program is evaluated, and the result joins an archive that informs the next proposal. The output of the search is an executable organization of model calls, prompts, feedback and control flow. It is more persistent than a single successful answer: it can be reused on new questions.

The paper introduces Automated Design of Agentic Systems, a broad research formulation, and Meta Agent Search, one particular algorithm within it. Keeping those names separate matters. ADAS can include many search algorithms and objectives. Meta Agent Search uses a largely fixed designer, an expanding archive, and task performance as its optimization signal. The paper proposes improving the designer itself as future work; the main experiments do not establish a recursively improving designer.

This distinction makes the paper useful background for self-evolving harnesses. A harness is the program that organizes a model’s interaction with tasks and tools. There are at least three different changes one might call improvement: improving an answer inside one execution, improving the reusable task-solving program, and improving the procedure that creates such programs. ADAS experimentally studies the second through a fixed instance of the third.

Figure 1. Original conceptual diagram based on Sections 2–3 and Appendix H. The search loop changes candidate programs and archive contents. The selected program later runs a separate task-solving loop. This is a distinction between learning levels, not a measured execution trace.

2. Prerequisites: programs, models and observations

2.1 A model call is a component, not the complete policy

Let a frozen foundation model be a stochastic mapping from a context to a response. A program can call that mapping repeatedly, choose which previous messages to include, invoke a tool, and decide when to stop. I write its output as

$$y=A(x;\theta,\omega),\qquad A\in\mathcal{S}. \tag{1}$$

Here $x$ is a task, $\theta$ denotes fixed model parameters, $\omega$ collects sampling randomness, and $\mathcal{S}$ is the allowed program space. Changing $A$ can change behavior even if $\theta$ never changes. A reflection loop and a vote among five answers are different programs around the same model.

This formalization is explanatory notation for the review, not a theorem or numbered equation claimed by the paper. The key modeling choice is to treat the program as the object being optimized. An agent’s name, such as “expert” or “committee,” has no independent force: its behavior depends on which information it receives and what it does with it.

2.2 There are two kinds of feedback

Task feedback concerns a particular answer. In ARC, candidate transformation programs can be checked against visible input-output demonstrations. Search feedback concerns an entire agent design. The evaluator runs that design across validation tasks and reports an aggregate score. A good result on one visible example is not the same as a good reusable design.

The distinction also separates error repair from optimization. A runtime exception tells the designer that a candidate could not complete its evaluation. A low score from a valid program says the design completed but solved too few tasks. They require different responses. Repairing an exception is not evidence of improved reasoning, while replacing a low-scoring valid design need not be a debugging operation.

2.3 Validation is a measurement channel

For $n$ tasks and $R$ repeated executions per task, a simple accuracy estimator is

$$\widehat{J}(A)=\frac{1}{nR}\sum_{i=1}^{n}\sum_{r=1}^{R} \mathbf{1}\{A(x_i;\theta,\omega_{ir})=y_i\}. \tag{2}$$

The indicator can be replaced by a task-specific score; DROP uses F1 rather than exact-match accuracy. Equation (2) makes explicit what the designer learns from: finite observations of a stochastic program, not access to its true quality on all future tasks. Repeating calls can reduce sampling noise, but cannot create new task diversity.

3. Why search in code space?

The paper decomposes ADAS into a search space, a search algorithm and an evaluation function. This sounds elementary, but it prevents a common confusion: a stronger optimizer cannot discover a behavior its representation cannot express. If the only editable field is a prompt, the system cannot introduce a new feedback loop unless the fixed surrounding program already supports one.

A graph representation offers more freedom. Nodes can represent model calls and edges can route their outputs. A program adds familiar conditional branches, loops, temporary memory and tool invocation. The paper chooses Python and gives the designer a small framework, so it mainly writes a forward function mapping task information to an answer.

Figure 2. Original comparison of representation choices discussed in Section 2. Code can express richer control flow, but the permitted interfaces and finite search budget still determine what can actually be reached.

Turing completeness is an argument about representational possibility under idealized resources. It does not mean a short search will find any useful agent. In practice the designer sees an API, formatting constraints, examples and an archive, and has a limited number of proposals. We can describe the distinction as

$$\mathcal{S}_{\mathrm{reachable}}(B,\theta,\mathcal{I}) \subseteq\mathcal{S}_{\mathrm{allowed}} \subseteq\mathcal{S}_{\mathrm{expressible}}, \tag{3}$$

where $B$ is the search budget and $\mathcal{I}$ is the interface. The three sets need not be equal. A program that requires unavailable tools is expressible but not allowed; an allowed program can still be too unlikely for the model to generate during the allotted search.

Providing a framework is therefore not a minor convenience. It concentrates search on useful variation instead of API boilerplate. Appendix C describes structured information records containing a name, author, content and iteration index, together with a model-call interface. That makes different modules easier to compose. The tradeoff is that the interface supplies a human-chosen vocabulary for the search.

My practical reading is that freedom and bias should be designed together. Restricting a search to safe, typed operations can improve proposal validity, while allowing all possible syntax can waste evaluations. A broader language is valuable when the task needs a genuinely new control pattern; a narrower graph may be preferable when the available operations are already sufficient and search efficiency dominates.

4. Meta Agent Search, step by step

4.1 The archive is the designer’s changing context

An archive entry contains a candidate design and its measured performance. Let $\mathcal{A}_t$ be the archive before round $t$. A compact description of the process is

$$A_t\sim q_{\theta_m}(\cdot\mid\mathcal{A}_t,d,\mathcal{I}), \qquad \mathcal{A}_{t+1}=\mathcal{A}_t\cup\{(A_t,\widehat{J}(A_t))\}. \tag{4}$$

The domain description is $d$, the designer model has fixed parameters $\theta_m$, and the proposal distribution changes because its context changes. This is a form of adaptation through accumulated information, not gradient training of the designer. The archive can carry both successful combinations and useful ideas that initially scored poorly.

The authors encourage proposals that are interesting relative to prior designs. In the demonstrated algorithm, that encouragement is expressed through the designer prompt and reflection. It is not a calibrated novelty distance or a formal guarantee that two differently named designs behave differently. The empirical score remains the main measured objective.

Algorithm 1. Meta Agent Search at the paper’s level of abstraction

The following is a paraphrase of Section 3 and Appendix H. It describes the method, including its two reflection passes and bounded error repair.

1  Receive validation tasks, a domain description and iteration limit N.
2  Initialize the archive with baseline designs, or an empty archive.
3  For each search round from 1 to N:
4      Propose an idea and a complete candidate agent program.
5      Reflect twice on novelty, correctness and design details.
6      Evaluate the candidate on the validation tasks.
7      If execution raises errors, refine and re-evaluate;
8          permit at most five such repair rounds.
9      Record the candidate and its evaluation information in the archive.
10 Return the final archive of discoveries.

The paper does not provide a complete specification of every terminal failure case in this short pseudocode. In particular, a production system should specify whether a design that still fails after the repair budget is stored as an invalid candidate or excluded from scored comparisons. I do not fill that gap by assuming an unreported rule.

4.2 Why retain stepping stones?

A design may introduce useful feedback diversity yet perform poorly because its answer selection is weak. Another design may have good selection but little diversity. An archive lets the designer combine these ideas later. Keeping only the highest-scoring candidate could remove useful components before their value becomes visible in a different composition.

Section 4.1 describes a progression in ARC from multiple initial solutions to refinement, more diverse feedback and combinations of feedback styles. The authors interpret this as innovation through stepping stones. I find the narrative plausible, but it is not a controlled estimate of the archive’s causal contribution. An archive-ablation study with matched evaluation budgets would be needed to isolate that contribution from more proposals, longer prompts or a lucky search trajectory.

The archive also has a scaling cost. If each candidate description occupies roughly $L$ tokens and every later round receives all earlier descriptions, then the cumulative archive material across $T$ rounds grows approximately as

$$L\sum_{t=0}^{T-1}t=\frac{LT(T-1)}{2}. \tag{5}$$

This is an accounting model, not a reported token measurement. It shows why an indefinitely growing archive will eventually need retrieval, compression or selective retention. Those changes can save context but also remove low-scoring stepping stones. Archive management is part of the search algorithm rather than an interchangeable storage detail.

5. Reading a discovered design as information flow

The ARC example clarifies what the search can produce. A task contains example grids and an unseen input grid. The task-solving agent proposes transformation code, tests that code on visible examples, gathers feedback and refines candidate transformations. The outer search evaluates whether this entire procedure generalizes across ARC tasks.

Figure 3. Original conceptual redrawing of Figure 3b and Appendix D. The visible examples belong to the task-solving loop. Hidden answers belong to evaluation. The diagram conveys the intended method without asserting that a particular program was executed here.

Algorithm 2. Conceptual ARC candidate-and-feedback procedure

This summary follows the paper’s illustrated design. Its “human-like” feedback is generated by a model, not by recruited human reviewers.

1  Generate five candidate transformation programs for the grid task.
2  Test candidates on the visible demonstrations.
3  Retain candidates that solve at least one demonstration.
4  Collect model feedback from several prompted perspectives.
5  Refine retained candidates for up to three rounds, checking examples.
6  Rank candidate transformations using the visible-example evidence.
7  Combine selected candidates into a final transformation.
8  Apply that transformation to the task's unseen input grid.

Three properties are worth separating. Candidate diversity increases the chance of proposing a useful rule. Feedback can expose a local defect in a rule. Final selection determines whether the system actually uses the good rule. Improving only the first property need not improve the answer if the selector is unreliable.

The paper also shows peer review, divide-and-conquer and a “Verified Multimodal Agent.” These names are generated design labels. The experimental framework uses text model calls, and the discussion identifies richer multimodal support as future work. Thus a generated name containing “multimodal” should not be read as evidence of an image encoder or a vision-model experiment.

For a task with $m$ candidate answers, suppose each is correct with probability $p$ and correctness events are independent. The chance that at least one is correct is

$$P(\text{some correct})=1-(1-p)^m. \tag{6}$$

This is an optimistic availability calculation. The final answer is correct only if the selection or combination step exploits that availability. Under a simplified selector that succeeds with conditional probability $s$ whenever a correct candidate exists, the final probability is $s[1-(1-p)^m]$. Increasing $m$ can be ineffective when $s$ falls because the selector becomes confused by extra conflicting evidence.

Real model samples are also correlated. Asking the same model five times does not guarantee five independent discoveries. Different personas can change wording while retaining the same misconception. This is why I would evaluate the diversity of errors and the quality of final selection separately, rather than treating the number of named agents as a measure of useful diversity.

6. What was actually evaluated?

The experiments use historical model snapshots. The designer is GPT-4o gpt-4o-2024-05-13, while most candidate agents and baselines use gpt-3.5-turbo-0125. Table 3 additionally transfers to Claude Haiku claude-3-haiku-20240307, the same GPT-4o snapshot, and Claude 3.5 Sonnet claude-3-5-sonnet-20240620. These are paper settings, not claims about current model rankings.

The ARC experiment is a deliberately restricted subset: tasks sampled from the public training collection with grid dimensions at most $5\times5$. There are 20 validation tasks and 60 test tasks, each evaluated five times. Search runs for 25 iterations. A result on this selected small-grid subset must not be presented as the score on the full ARC benchmark or a competition evaluation.

For the other domains, search runs separately for 30 iterations. DROP measures reading comprehension using F1; MGSM measures multilingual mathematics; MMLU covers multiple subjects; GPQA uses difficult science questions. Appendix E gives 128 validation and 800 test questions for each non-GPQA domain. GPQA Diamond uses 32 validation and 166 test questions, with five repeated evaluations; the other domains use one. DROP uses one-shot questions, while the remaining domains use zero-shot style questions.

Setting	Validation tasks	Test tasks	Repeats per task	Search rounds
Restricted ARC subset	20	60	5	25
DROP, MGSM, MMLU, each	128	800	1	30
GPQA Diamond	32	166	5	30

This table describes task counts, not independent draws from all possible tasks. The authors use repeated executions to stabilize scores, but a repeated evaluation of the same question shares its difficulty and content with the other repeats. That distinction becomes important when interpreting uncertainty and search feedback.

Baselines include chain-of-thought, self-consistency, self-refinement, debate and a quality-diversity-inspired procedure. Reasoning experiments add step-back abstraction and role assignment; prompt optimization is a separate automated comparison. The paper implements baselines in the same framework. That improves interface consistency, but does not by itself match model calls, tokens or latency across all designs.

7. Domain-specific results: where is the gain?

Figure 4. Replot of Table 1. Bars show the strongest manual baseline in each column, prompt optimization, and Meta Agent Search. Error bars reproduce the reported 95% bootstrap intervals. DROP is F1 on a 0–100 scale; the other tasks use accuracy percentages.

The large gains are in reading comprehension and multilingual mathematics. DROP rises from the best manual result of 65.8 to 79.4 F1. MGSM rises from 39.0 to 53.4 accuracy. MMLU rises from 65.9 to 69.6, and GPQA from 31.6 to 34.6. Subtracting those task-specific reference scores gives

$$\Delta_{\mathrm{manual}}=(13.6,\ 14.4,\ 3.7,\ 3.0). \tag{7}$$

These are score-point differences, with an F1-point difference for DROP and percentage-point differences for accuracy. They are not relative percentage increases. For example, 53.4 minus 39.0 is 14.4 percentage points; dividing by 39.0 would answer a different question. I do not average these four numbers because the metrics and tasks are different.

The prompt-optimization comparison is also informative: its scores are 69.1, 30.6, 67.6 and 32.9. Code-space search can change more than instruction wording, and the results support the value of that freedom in this setup. They do not isolate which particular additional freedom caused the improvement. Search budget, design complexity and inference-time computation can all matter.

The authors hypothesize that knowledge availability limits gains on MMLU and GPQA, while more of the errors in DROP and MGSM can be repaired by better organization. This is a useful hypothesis, not an experimentally identified decomposition of errors. A stronger test would distinguish missing knowledge, incorrect reasoning, retrieval failure and answer-selection failure at the task level.

The reported confidence intervals should not be treated as a shortcut for pairwise significance testing. Overlap or non-overlap of marginal intervals does not replace a paired comparison on identical tasks. The smaller GPQA improvement deserves particular caution because it is only three points and the reported intervals are several points wide.

8. Transfer is real, but it is not uniform

8.1 Across domains

Table 2 transfers three designs found on MGSM to other tasks. Dynamic Role-Playing reaches 69.5 on GSM8K and 31.2 on GSM-Hard. Compared with the strongest manual score in each column, those are gains of 25.9 and 13.2 percentage points. The pattern is consistent with learning reusable reasoning organization rather than merely memorizing a few MGSM answers.

The broader appendix is essential. Table 5 shows that transfer to GPQA does not improve the best manual point estimate. The three transferred scores are 28.6, 28.7 and 29.9, below 31.6. On MMLU, one transferred design reaches 67.0, while the other two reach 62.4 and 64.8, below the manual reference of 65.9. It would be incorrect to say that every discovered design improves every destination.

Figure 5. Differences computed from Tables 2 and 5 relative to the best manual point estimate in each destination. Positive values are gains in score points. This heatmap exposes the negative transfers as well as the gains; it is not a significance map.

The distinction between “there exists a transferable design” and “the selected design will transfer” is crucial for deployment. If we choose the destination’s best design after viewing destination scores, we are performing additional selection. A user who has only source-domain validation results cannot automatically reproduce that oracle choice.

8.2 Across model backends

Table 3 changes the model used by ARC agents. The structured-feedback design is strongest among the three on GPT-3.5 at 13.7, but dynamic memory is stronger on GPT-4o at 37.0 and Claude 3.5 Sonnet at 48.3. Model capability changes which organization is useful. A complicated critic structure can help a weaker model yet be less useful than a simpler refinement structure around a stronger model.

Figure 6. Table 3 point estimates, with the best manual baseline chosen separately for each backend. Lines connect categorical model settings for readability; they are not a scaling curve. The winning architecture changes with the model.

The strongest headline, 48.3 on Sonnet, belongs to the restricted ARC experiment described earlier. There are also exceptions within the table: structured feedback scores 5.0 on Haiku against a manual best of 6.3, and 38.7 on Sonnet against 39.3. The family of searched designs transfers well in several settings; not every member dominates.

Section 4.3 says the top three ARC agents were selected using their GPT-3.5 test accuracy before transfer. That makes the selection rule part of the interpretation. It does not automatically mean target-model scores were used for selection, but it does mean the source test result is not a pristine final report for a single validation-chosen architecture. A stronger deployment claim would predefine selection on validation tasks and reserve an untouched final test.

9. Initialization: useful priors can also narrow exploration

The standard search initializes the archive with human-designed baselines. Appendix I compares an empty archive. The result is not a monotone win for seeding: DROP improves with seeds, but MGSM is substantially higher with empty initialization, 67.5 instead of 53.4. MMLU and GPQA show smaller seeded advantages.

Figure 7. Replot of Table 6, including reported 95% intervals. The empty-archive condition still uses a pretrained designer, the framework and domain instructions. It removes explicit seed designs, not all prior knowledge.

The math result changes how I interpret the archive. It is a source of reusable ideas and also an anchor that can bias proposals toward known patterns. A useful default can become a constraint on exploration. Under a small number of rounds, these effects can compete strongly.

The authors suggest that broader exploration explains the math advantage. That explanation is plausible but unproven by one table. To test it, I would compare several search runs under seeded, empty and mixed initialization, recording unique control-flow patterns, candidate cost and score trajectories. An empty start could win because of a broader proposal distribution, a lucky early candidate, or interactions with the prompt length. Those explanations imply different remedies.

It is also important not to silently substitute the 67.5 result into Table 1’s 53.4 when discussing that table’s setup. They correspond to different initialization conditions. The interesting conclusion is that initialization is an important algorithmic variable, not that a single best number describes the entire method.

10. How noisy feedback can mislead search

10.1 Repeats and task diversity solve different problems

Consider repeated binary outcomes on one task, with variance $\sigma^2$ and equal pairwise correlation $\rho$. The average of $R$ repeats has variance

$$\operatorname{Var}(\bar{Z}) =\frac{R\sigma^2+R(R-1)\rho\sigma^2}{R^2} =\frac{\sigma^2}{R}\,[1+(R-1)\rho]. \tag{8}$$

The first equality counts $R$ variance terms and $R(R-1)$ covariance terms. Compared with independent repeats, the effective repeat count is therefore

$$R_{\mathrm{eff}}=\frac{R}{1+(R-1)\rho}. \tag{9}$$

For five repeats and hypothetical $\rho=0.5$, the equivalent is about 1.67 independent repeats. This is an illustrative calculation, not an estimated correlation in ADAS. It explains why five runs of 20 ARC validation questions do not provide the same evidence as 100 genuinely different questions.

Figure 8. Original analytical illustration of Equation (9). Correlations are hypothetical. Repetition helps with stochastic execution, while new tasks help with diversity; neither substitutes fully for the other.

For a fixed candidate on $n$ independent bounded task scores, Hoeffding’s inequality gives a useful reference:

$$P(|\widehat{J}-J|\ge\epsilon)\le2\exp(-2n\epsilon^2). \tag{10}$$

If $K$ candidate programs are fixed independently of the evaluation data, applying a union bound yields

$$P\!\left(\max_{k\le K}|\widehat{J}_k-J_k|\ge\epsilon\right) \le2K\exp(-2n\epsilon^2). \tag{11}$$

Solving the right side for a confidence level $\delta$ gives $\epsilon=\sqrt{\log(2K/\delta)/(2n)}$. This derivation shows why considering more alternatives increases the chance of an overly optimistic winner. It is not a guarantee for Meta Agent Search: its candidates adapt to reused validation feedback, violating the fixed-candidate assumption. I include the bound to expose the assumption that a simple calculation would otherwise hide.

10.2 A score is an information bottleneck

Two designs can both score 50% while failing on disjoint halves of a dataset. The archive’s aggregate metric cannot reveal whether one can complement the other. A runtime trace could distinguish bad decomposition from bad aggregation, but rich trace analysis is identified as future work rather than established in the main procedure.

A better evaluator might return categorized errors or a small set of contrasting examples. That could improve sample efficiency because the designer learns why a design failed. It could also create additional overfitting if the same examples are exposed repeatedly. Richer feedback does not remove the need for an independent evaluation boundary.

11. Cost, latency and the actual optimization target

Appendix J reports approximately US$500 for an ARC search-and-evaluation run and US$300 for a reasoning-domain run at the historical API prices. The authors identify evaluation calls as the main expense. Those amounts are historical experimental observations, not an estimate for running the same search today.

I find it useful to split total development and usage cost:

$$C_{\mathrm{total}}=C_{\mathrm{search}}+M\,C_{\mathrm{serve}}, \tag{12}$$

where $M$ is the number of future tasks. If a searched design is cheaper per task than a reference by $\Delta C>0$, search cost is recovered in monetary terms once

$$M>\frac{C_{\mathrm{search}}}{\Delta C}. \tag{13}$$

ADAS does not demonstrate that its best-scoring designs are cheaper. They may use more calls, so Equation (13) may not apply at all. A more expensive design can still be valuable when accuracy has sufficient utility, but that is a different tradeoff.

Latency depends on dependencies, not just call count. Ten independent calls can run concurrently if resources permit; ten sequential refinements cannot. For a directed execution graph with call times $\ell_v$, an idealized lower bound is the longest dependency path:

$$L_{\mathrm{critical}}=\max_{\pi}\sum_{v\in\pi}\ell_v. \tag{14}$$

Queueing, rate limits and tool time can make real latency larger. A search that optimizes only accuracy has no reason to prefer a short critical path. A fair engineering evaluation would compare the best attainable accuracy at matched token, cost or latency budgets, rather than assume that a higher benchmark score is a free systems improvement.

The paper explicitly lists multi-objective ADAS as future work. A scalar objective such as $J-\lambda C$ is one option, but the choice of $\lambda$ encodes an application preference. A Pareto set of designs can be more useful: one user may need minimum latency, another may accept slow reasoning for a valuable task. The appropriate design is not necessarily the highest-scoring archive entry.

12. Design alternatives and failure boundaries

Keep only the incumbent or preserve an archive? An incumbent is cheap to describe and easy to deploy. An archive supports recombination and avoids forgetting useful partial ideas. The archive becomes less helpful if it is redundant, stale or too long for the designer to use. Retrieval based only on score may discard novel stepping stones, while unrestricted retention may overwhelm context.

Unrestricted programs or structured modules? Programs make loops and tool use natural, but increase the probability of invalid or resource-intensive proposals. Structured modules restrict behavior and may make search more efficient. The right choice depends on whether the task needs new operations or merely a better arrangement of known ones. Expressiveness alone is not an empirical win.

Several critics or an external check? Prompted critics provide diverse verbal feedback, yet can share the generator’s misconception. A deterministic task check supplies a different kind of evidence, but may verify only the visible examples. In ARC, passing those examples does not uniquely establish the transformation rule on the unseen input. A critic and a test are complementary signals with different failure modes.

More refinement or early stopping? Refinement can correct mistakes but can also corrupt a correct answer. Stopping when a model says “correct” makes success depend on self-assessment calibration. A bounded loop protects the budget, but a bound chosen for one backend may be unsuitable for another. Table 3’s changing architecture ranking is a reminder that these choices are model-dependent.

Search on one domain or several? Specialization yields a clear evaluation signal and can exploit recurring task structure. Multi-domain search can encourage generality but introduces conflicting objectives and sampling choices. An aggregate score may conceal a large loss on a small domain. A robust objective would make those tradeoffs visible instead of hiding them in an average.

13. Limitations of the evidence

The paper establishes useful results in specific question-answering setups. It does not establish reliable multi-day software work, continual adaptation in a changing environment, or autonomous repair of a deployed harness across arbitrary tasks. Those scenarios require persistent state, feedback delays and failure recovery beyond the evaluated single-task settings.

The number of search rounds and validation tasks is modest. Search outcomes can be sensitive to early proposals and initialization. The reported task-level intervals do not, by themselves, quantify variability over independent complete searches. A narrow interval around a discovered design’s test score is not the same as confidence that a new search will discover a similarly good design.

The interpretation of transfer requires selection details. The paper’s explicit use of source test accuracy to choose ARC designs for transfer should remain visible. Reported improvements also do not isolate architecture from additional inference-time effort. The same framework is a helpful control, but a common interface is weaker than a common compute budget.

Several names and broad prose claims need careful reading. “Human-like” critics are model prompts. “Multimodal” in an agent label does not establish a multimodal backend. Appendix G describes one domain as reading comprehension while naming GPQA; elsewhere GPQA is consistently the science task and DROP is reading comprehension. I rely on the experimental definitions and tables rather than infer an extra experimental condition from that wording inconsistency.

Finally, learned programs can produce unexpected behavior. Section 6 discusses isolated execution and other precautions. My main methodological concern here is that changing the executable search object also changes how it can exploit the evaluation environment. An evaluation protocol must distinguish legitimate task solving from dependence on unintended information or unconstrained resource use. This is a property of the experimental design, not evidence that a particular exploit occurred.

14. Independent critical analysis

14.1 The strongest contribution is a reusable experimental unit

For me, the durable contribution is not a particular committee or critic prompt. It is treating the reusable task-solving program as a candidate that can be generated, evaluated and retained. That turns agent design into an explicit search problem and allows control flow to become a learned artifact. The framework also preserves a legible object that researchers can reason about at the algorithmic level.

The evidence is strongest for the claim that this search can discover useful designs in the tested domains. It is weaker for the causal claim that archive-driven novelty, rather than extra proposals or extra calls, explains the gain. It is weaker still for open-ended improvement of the search procedure itself. These are distinct hypotheses and deserve distinct experiments.

14.2 Selection is part of the algorithm

Suppose a system produces 30 architectures and we report the destination task’s best one. That result answers an existence question. To answer the practical question, we must also explain how a user with no destination test labels selects the architecture. The gap between those questions becomes especially clear in the negative cells of Figure 5.

I would therefore evaluate a frozen selection policy, not just a portfolio of candidate scores. The policy might choose by source validation performance, a small destination calibration set, or a cost-constrained score. Whichever rule is used, its information access should be fixed before final testing. Otherwise, the selection procedure can quietly become a stronger component than the published search algorithm.

Figure 9. Original recommendation for separating search feedback, design selection and final evaluation. This is a proposed follow-up protocol, not a claim that the paper used three separate datasets in every experiment.

Algorithm 3. A proposed follow-up experiment for transferable designs

This is my evaluation recommendation, distinct from Algorithm 1.

1  Fix task splits, model snapshots, allowed tools and resource budgets.
2  Run several independent searches using only search feedback.
3  Select one design per search using a declared selection rule.
4  Freeze the selected designs and their stopping conditions.
5  Evaluate on fresh tasks and declared destination models.
6  Report per-task quality, cost, latency and negative transfers.
7  Separate variation across tasks from variation across searches.
8  Compare seeded, empty and restricted-space search at matched budgets.

14.3 Better performance and better improvement are different

An agent that answers more questions correctly is better at the task. A designer that discovers equally good agents with fewer evaluations is better at improvement. A recursively improving system would need to demonstrate the latter level changing productively, rather than relabel changes to the archive as changes to the designer.

This suggests a useful follow-up metric: quality achieved as a function of total search cost, evaluated on fresh tasks. The object of comparison becomes an improvement process, not only its single best output. A meta-harness should earn its name by improving that curve under controlled information access and budgets.

I would also want a stability test. After a model backend changes, can a limited amount of feedback select or adapt the harness without losing previously solved cases? Table 3 shows that architecture rankings change. A system that notices such changes and updates its selection policy would address a practical need not resolved by a one-time transfer table.

15. Conclusion

ADAS makes a simple but consequential move: the program around a language model becomes a search object. Meta Agent Search demonstrates an accessible way to propose those programs, use validation feedback, and carry previous discoveries forward in an archive. The large gains on DROP and MGSM, the successful transfers and the nontrivial initialization result make the idea worth studying.

The strongest reading keeps its boundaries intact. Code-space expressiveness is not a search-efficiency guarantee. Repeated task executions are not new tasks. A family of transferable designs is not a guarantee that every design transfers. And a fixed designer that improves task-solving programs is not yet an experimentally demonstrated self-improving designer.

For subsequent harness research, I would retain three questions from this paper: what can change, what feedback drives the change, and how is the final design selected without using the answer to the evaluation? Those questions are more useful than counting agents or attaching increasingly ambitious names to the same loop.

References and figure notes

1. Shengran Hu, Cong Lu, Jeff Clune. Automated Design of Agentic Systems, arXiv:2408.08435v2. Main source; 34 pages including Appendices A–J. Sections 2–3 define the formulation and search; Sections 4.1–4.3 and Tables 1–6 supply the experiments; Appendices D–F give task settings; Appendix J gives historical cost.
2. Author project page. Resource and figure context.
3. Official ADAS resource. Resource pointer.

Figures 1–3 and 9 are original conceptual diagrams. Figures 4–7 replot explicitly identified paper tables, retaining metric distinctions and negative results. Figure 8 is an original analytical illustration under stated assumptions. Equations formalize the review’s reasoning; they are not attributed as the paper’s numbered equations. No new model benchmark or reproduction result is claimed.