Darwin Gödel Machine: How a Coding Agent Learns to Improve Its Own Harness

Review date: October 4, 2026
Author: Zhongzhu Zhou
Paper reviewed: Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents
Paper authors: Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, Jeff Clune
arXiv: 2505.22954v3, March 12, 2026

1. What changes when the agent can edit itself?

The most useful way to read the Darwin Gödel Machine (DGM) is to separate the language model from the software that lets it work. The model supplies suggestions, code, and reasoning. The surrounding program decides which tools exist, how tool results enter context, when to try again, and how to choose between proposed solutions. DGM searches over that surrounding program. Its central question is whether a coding agent can improve this program and then use the improved agent to make the next change.

My main takeaway is that the paper makes a credible case for automated accumulation of useful agent designs. It does not establish unlimited recursive improvement, prove that each change is beneficial, or demonstrate training a better foundation model. The foundation models stay frozen. The archive manager and parent-selection process also stay fixed. These boundaries matter because the attractive phrase “self-improvement” otherwise bundles together several very different capabilities.

The paper reports a change from 20.0% to 50.0% on its 200-task SWE-bench evaluation, and from 14.2% to 30.7% on the full Polyglot benchmark. The first pair is not a result on all 500 SWE-bench Verified tasks. Polyglot’s headline full-benchmark result must also be distinguished from the 14.0% to 38.0% result on the 50-task search subset. I will keep these populations separate throughout this review.

There are three connected ideas. First, an agent is an editable artifact rather than an immutable wrapper. Second, the improved artifact participates in later editing, so the search operator can change along a lineage. Third, a persistent archive retains multiple lineages, including agents whose immediate scores are lower than their parents. An initially disappointing tool change may become useful after another change. Deleting every such intermediate version would prevent the search from discovering that combination.

Figure 1 (paper Fig. 1): DGM alternates between modifying an agent and evaluating its ability to solve external coding tasks. Source: Zhang et al., arXiv v3, p. 2. The archive holds distinct agent versions; it is not a conversation memory containing only textual reflections.

The review below uses the paper and its appendices as evidence. Numerical examples and statistical calculations are explicitly labeled as my analysis. I have not run the paper’s experiments. In particular, the figures reproduced from the paper show reported observations, while the selection-weight and uncertainty plots are explanatory calculations.

2. Prerequisites: three levels of change

2.1 Model weights, context, and the agent program

Let a frozen model have parameters $\theta$, let $g$ denote an agent program, and let $x$ be a task. A task execution produces an action trajectory and a final patch. For exposition, I write its distribution as

$$\tau \sim P(\tau\mid x,g,\theta),\qquad y=\operatorname{Output}(\tau). \tag{1}$$

This is notation for the review, not an equation claimed by the authors. It makes a simple point: changing $g$ changes behavior even if $\theta$ never moves. Replacing whole-file editing with a precise replacement tool alters the actions the model can express. Summarizing old messages changes the information on which subsequent actions depend. Asking a second model to rank patches changes the output selection rule.

Changing context inside one execution is not the same as persisting a new agent program across tasks. A reflection can improve the next attempt on the same issue without teaching the next issue anything. DGM persists modifications to $g$, so the next evaluation starts with a different reusable system. It remains possible for the new system to be worse, even when the modification sounds sensible.

These distinctions also clarify what “learning” means here. No gradient update is required for the overall system to learn a useful design. But calling the process learning should not imply that the base model has acquired new internal knowledge. The persistent object is executable agent software. The measured gain belongs to the combination of model, program, tools, and evaluation conditions.

2.2 Empirical search versus proof of improvement

A classical Gödel-machine proposal seeks justified self-rewrites using formal proofs relative to a specified utility and assumptions. DGM replaces that burden with empirical evidence from coding benchmarks. That makes the procedure usable with current models, but removes the corresponding guarantee. A finite benchmark score is an estimate under one distribution; it is not a proof that the agent is better on every future problem.

There is a second subtlety. DGM does not demand a strictly better score before keeping every child. Its archive admits agents that remain functional, even if they score below their parents. “Validate empirically” therefore means measure and filter candidates, not certify monotonic improvement of each lineage. The best score in an expanding archive can rise monotonically while individual descendants regress.

This is a productive tradeoff when useful changes interact. A new context-management mechanism might initially discard useful information, then become valuable after the agent learns when to invoke it. Conversely, storing a weak child costs evaluation and future search budget. The archive makes delayed benefits possible; it does not make every detour worthwhile.

2.3 Why an archive differs from keeping the latest version

A useful analogy is a laboratory retaining several prototypes. The current fastest prototype may be hard to modify. A slower one may expose a cleaner interface that makes a future improvement easier. If the laboratory keeps only the latest prototype, one poor redesign can reduce both current performance and its ability to recover. If it keeps only the best prototype, it can repeatedly miss a route that begins with a small regression.

DGM’s archive contains programs, scores, and ancestry. Its notion of exploration should not be confused with a formal guarantee of behavioral diversity. The reported selection rule uses performance and the number of functional children. It does not directly measure how different two agents’ behaviors are. Several syntactically different programs could implement effectively the same strategy, yet remain separate archive entries.

The distinction between preservation and selection is important. Preserving an agent gives it a chance to be revisited. Selection allocates actual computational effort to that chance. An archive can grow indefinitely in principle while individual entries become so unlikely to be selected that they are effectively abandoned within any realistic budget.

3. The actual self-improvement loop

3.1 What the initial agent can do

The starting agent has a shell tool and a basic editor. The editor can inspect a file, create a file, or replace an entire file; it initially lacks the finer-grained editing interface later discovered during search. The task prompt identifies the repository, describes the problem, and supplies testing information. The model decides which tool to invoke.

This starting point is intentionally lightweight. It creates room for large gains from ordinary interface improvements. That is a reasonable experimental choice for demonstrating the mechanism, but it changes how I interpret the size of the result. Improving a minimal editor is easier than proving that the same process materially improves an already mature coding harness. A strong starting system is a necessary next experiment, rather than a reason to dismiss the reported result.

For a selected parent, the system uses evaluation traces to formulate a proposed change, then asks the parent agent to implement it in its own program. Appendix C.3 makes an important division of labor explicit: a separate o1 model diagnoses improvement opportunities. The selected coding agent implements the proposal. Thus the experiment has a changing implementation agent, a separate diagnostic model, and a fixed outer search controller. It is more precise to describe this as self-modification with external diagnosis than as a system inventing every part of its own improvement process.

3.2 A compact mathematical description

Let $A_t$ be the archive after step $t$, $L(g)$ the evaluation logs of agent $g$, and $D$ the diagnostic process. Let $M_g$ denote the editing procedure executed using agent $g$. Then the paper’s mechanism can be summarized as

$$\text{proposal}_t=D(L(g_t),g_t),\qquad g'_t=M_{g_t}(g_t,\text{proposal}_t). \tag{2}$$

The important dependency is the subscript on $M$. In DGM, the parent participates in producing its child. In the baseline without self-improvement, the editor is always the initial agent, so that dependency becomes $M_{g_0}$. Both methods can improve the target agent. Only the former changes the agent that carries out later modifications along a lineage.

After evaluation, the archive update is

$$A_{t+1}=\begin{cases} A_t\cup\{(g'_t,\widehat\alpha(g'_t))\}, & V(g'_t)=1,\\ A_t, & V(g'_t)=0. \end{cases} \tag{3}$$

Here $V$ represents the paper’s functional admission condition, including successful compilation and retained code-editing ability. The score $\widehat\alpha$ is a measured success fraction. Notice that equation (3) contains no requirement that the child beat its parent. This is where a functioning but temporarily weaker stepping stone can survive.

3.3 Algorithm 1: archive-based self-modification

01  Start with the initial coding agent and its evaluation record.
02  Keep the model parameters and outer search rules fixed.
03  Select eligible parent agents from the current archive.
04  For each selected parent, obtain a diagnosis from its logs.
05  Convert the diagnosis into one proposed agent improvement.
06  Let the selected parent edit a copy of its own program.
07  Evaluate the resulting child using the staged protocol.
08  Discard children that do not retain basic editing functionality.
09  Archive functional children, their scores, and parent links.
10  Repeat until the specified generation budget is exhausted.
11  Report the selected agent on the stated evaluation population.

This is explanatory pseudocode reconstructed from Section 3 and Appendix C; the implementation-specific diagnosis and evaluation stages are made explicit. The paper describes 80 generated-agent iterations, with two jobs in parallel for SWE-bench and four for Polyglot. Its appendix also expresses the process as rounds selecting multiple parents. I treat the displayed iteration axis as the reported search budget and do not silently multiply 80 by the parallelism factor.

The distinction matters for cost comparisons. Two methods can have the same number of generated candidates but very different total token usage, evaluation coverage, and elapsed time. A candidate that fails the initial screen is cheaper than a successful candidate evaluated on 200 tasks. Parallelism primarily changes elapsed time and API contention; it does not erase the underlying evaluation cost.

4. Parent selection, derived step by step

4.1 Turning scores into sampling probabilities

Appendix C.2 provides a concrete selection rule. First form an eligible set $E_t$ containing archive agents whose reported score is below one. For an eligible agent $i$, let $\alpha_i$ be its score and $n_i$ its number of children retaining code-editing functionality. Scores are transformed with a sigmoid:

$$s_i=\frac{1}{1+\exp[-\lambda(\alpha_i-\alpha_0)]}, \qquad \lambda=10,\quad\alpha_0=0.5. \tag{4}$$

The transform puts an agent scoring 0.5 at weight 0.5 before accounting for children. Scores below 0.5 receive smaller but positive weights. Scores above 0.5 approach a ceiling of one. Compared with a hard threshold, the sigmoid leaves weaker candidates eligible. Compared with using raw accuracy, it changes the relative preference for candidates near the midpoint.

Next discount a parent that has already produced many functional children:

$$h_i=\frac{1}{1+n_i},\qquad w_i=s_i h_i, \qquad p_i=\frac{w_i}{\sum_{j\in E_t}w_j}. \tag{5}$$

I write the denominator explicitly over the eligible set. The appendix uses archive indexing in the denominator while describing sampling over eligible agents; expressing eligibility in the sum avoids ambiguity about perfect-score entries. Sampling is with replacement, so a parent can be drawn more than once in a batch.

The factor $h_i$ is called a novelty bonus in the paper, but it measures something narrower: how many functional children a parent already has. It is not the novelty of the parent’s algorithm, the uniqueness of the child’s behavior, or the number of attempted edits. A parent with many unsuccessful attempts may still have a small $n_i$. That choice can maintain exploration, but it can also allocate effort repeatedly to a difficult lineage.

Figure 2 (original calculation): Unnormalized weights from equations (4)–(5), using the paper’s parameters. More functional children reduce a parent’s sampling weight. The curves are analytical illustrations, not measured DGM trajectories.

4.2 A numerical example

Consider three hypothetical eligible agents. A has score 0.5 and four functional children; B has score 0.4 and none; C has score 0.2 and none. The sigmoid values are approximately 0.5000, 0.2689, and 0.0474. Their weights are therefore 0.1000, 0.2689, and 0.0474. Normalizing gives

$$(p_A,p_B,p_C)\approx(0.2402,0.6459,0.1139). \tag{6}$$

The best-scoring parent is not the most likely parent. B gets most of the probability because its lineage has not yet produced functional children. C still receives about eleven percent in this three-agent illustration. The example shows why the method can leave a currently strong branch and revisit a weaker alternative.

For a batch of $k$ independent draws with replacement, the probability of selecting agent $i$ at least once is

$$P_i(\text{selected at least once})=1-(1-p_i)^k. \tag{7}$$

For the hypothetical C above, this is about 21.5% at $k=2$ and 38.4% at $k=4$. These are calculations for a fixed distribution. In a real run, the archive, child counts, and scores change, so repeated rounds cannot be modeled by blindly reusing one constant probability.

4.3 Why positive probability is not a finite-budget guarantee

Suppose a useful lineage has probability $p$ of being chosen on each draw and that this probability stays constant. The chance that it is never selected after $K$ draws is $(1-p)^K$. If $p$ is only 0.001, 80 draws leave a roughly 92.3% chance of never visiting it. This illustrative calculation shows why “every agent has nonzero probability” should be read as a property of the search distribution, not as evidence that a modest run explores all useful paths.

Archive growth may dilute a lineage’s probability further. Performance noise can also move a promising agent into an unfavorable part of the sigmoid. An alternative would reserve a fixed exploration fraction for under-tested agents or use uncertainty-aware selection. Such policies introduce their own assumptions, especially when scores come from different task subsets. The paper’s rule is appealing because it is simple; its simplicity is not evidence of optimal compute allocation.

5. Stepping stones and what actually improved

The archive tree is the most informative figure in the paper. The final best agent descends through points where the measured score decreases. A policy that accepted only immediately improving children could remove those paths. The tree also shows branching from intermediate versions, which allows alternative implementations of a similar idea to coexist.

Figure 3 (paper Fig. 3): The SWE-bench archive and progress of the final lineage. Source: arXiv v3, p. 7. Node outlines distinguish evaluation coverage, so the diagram should not be read as though every node had the same measurement precision.

The named improvements are concrete: reject empty patches and retry; inspect a range of file lines; replace a unique string rather than regenerate a whole file; summarize context when necessary; generate several candidate patches; and use the history of previous candidates when producing later ones. These are changes to the interface and control flow of the agent. They do not require the model to acquire a new programming concept through weight training.

A finer-grained editor helps because it reduces the surface area of an action. If a task needs a three-line change, rewriting a thousand-line file exposes far more opportunities for omissions. A context summary helps for a different reason: it preserves access to recent evidence after the raw interaction history grows too long. It can also fail by discarding a subtle constraint. Patch ranking can improve final selection but introduces a new judgment problem: the ranking model may prefer a plausible explanation over a correct patch.

Appendix G compares two different realizations of finer-grained editing and reports markedly different agent scores, 23.3% and 40.5%. This supports a useful engineering lesson: a high-level feature label does not uniquely specify a successful agent interface. However, the appendix’s displayed suggestion labels and narrative mapping are not fully aligned. I therefore use the comparison to motivate interface sensitivity, without treating it as a clean causal estimate of one isolated edit primitive.

5.1 Why tool improvements can compound

A better editor can improve both external task solving and future self-modification, since both require making code changes. This shared capability creates a plausible feedback channel. It is stronger than an example where a system improves a text-answering prompt but relies on an unrelated fixed optimizer to make all future changes.

Nevertheless, shared activity is not equivalence of objectives. Repairing a Django bug and improving a model’s tool interface both involve code, but require different understanding. An agent could specialize in repository fixes without becoming better at experimental design, failure diagnosis, or selecting valuable changes. The paper assumes a useful relationship between benchmark coding skill and self-improvement skill. The relationship is plausible and partly supported by the functional-child results, but it is not a theorem.

For clarity, define $q(g)$ as expected external task quality and $r(g)$ as expected quality gain obtained by using $g$ to create a child. The desired relationship is positive association between $q$ and $r$. Observing an increase in $q$ alone does not determine $r$. A direct measurement would hold a set of parent programs and improvement proposals fixed, then compare children produced by the initial editor and by improved editors under equal budgets. That would isolate the part of recursion that is otherwise mixed with changing targets and diagnoses.

5.2 Three distinct counterfactuals

The main ablations remove different parts of the mechanism. The fixed-editor baseline still has an archive but always uses the initial agent to produce edits. The latest-only baseline changes the editing agent but retains only the latest functional version. The greedy baseline in Appendix A.3 retains access to past versions but always branches from the best-scoring node.

These are not interchangeable controls. Latest-only is vulnerable to regressions that replace a good current version. Greedy avoids that specific failure, but can reject a useful route through a temporarily weaker parent. Full DGM can preserve and resample such parents. Comparing only with latest-only would leave open the possibility that a simpler best-so-far strategy explains the benefit. The greedy ablation makes the archive argument substantially stronger.

6. Evaluation: the denominator is part of the result

6.1 The two task families

SWE-bench Verified contains repository-level Python maintenance problems. Polyglot contains programming tasks across several languages and typically requires a solution concentrated in a single file. The paper uses both because they put different demands on repository navigation, tool use, and reasoning. A gain on one does not automatically imply a gain on the other.

In the Polyglot evaluation, the authors use pass@1 without ground-truth test feedback during task solving. This differs from the benchmark leaderboard’s pass@2 setup, which permits one round of feedback from hidden tests. The comparison with Aider is therefore made under the paper’s adjusted setting. One must not compare the DGM pass@1 number directly with an arbitrary leaderboard pass@2 number and call the difference a system advantage.

“Pass@1” also does not imply one model call or no internal search. An agent may generate and rank several internal candidates before returning one final answer. The defining constraint here concerns how final task success is assessed and whether hidden-test feedback is revealed, rather than a universal bound on internal computation.

6.2 Staged evaluation and a documented ambiguity

The first SWE-bench screen uses ten tasks to assess basic functionality. Functional agents proceed to another fifty tasks, producing the sixty-task score used for screening. Strong candidates are expanded to 200 tasks, combining the previous sixty with 140 additional tasks. The stated trigger is a score over 40% and placement among the archive’s top two; the figure legend uses a greater-than-or-equal sign. That boundary difference should be resolved when specifying an experiment, rather than concealed by an overly precise reconstruction.

Polyglot is described using an initial ten-task screen and a fifty-task evaluation subset. Section 4.4 explicitly calls the plotted search result a 50-task result, while Appendix E.3 lists ten initial tasks and fifty additional task names, including an overlapping task. I preserve the authors’ reported 50-task label for the plotted scores and do not infer a clean sixty-distinct-task pool. The full-benchmark evaluation is the clearer evidence for the final 14.2% to 30.7% improvement.

6.3 Algorithm 2: read a staged result correctly

01  Record which tasks were used for the functionality screen.
02  Separate screening scores from expanded evaluation scores.
03  Record the promotion threshold and ranking condition.
04  Preserve task identities and overlaps between stages.
05  Identify which scores influenced later search decisions.
06  Select the final agent without using an untouched test set.
07  Evaluate that selected agent once on the untouched population.
08  Report its denominator, model, feedback access, and cost.
09  Keep subset results and final test results in separate columns.

Steps 1–5 explain the information needed to interpret the paper; steps 6–9 express my recommended reporting discipline. They are not a claim that the paper used a fully untouched final test set for every number. The SWE-bench search uses its 200-task evaluations in identifying strong candidates, so that score should not be described as a clean final holdout.

6.4 Why the ten-task screen can distort the search

The initial SWE-bench functionality tasks listed in Appendix E.2 are all Django tasks. That is an inexpensive way to discover obviously broken agents, but functional survival is then coupled to one framework and a small task set. A candidate that is useful on another ecosystem could be screened out before that strength is measured. Conversely, a candidate especially compatible with the screen could receive more downstream evaluation opportunities.

This is a selection effect, not proof that the reported result is invalid. A better design would separate a task-independent editing-capability test from a stratified task-performance screen. The former would answer whether the agent can still carry out the required operations; the latter would estimate quality across a representative mix of repositories. Conflating them makes the admission rule harder to interpret.

7. Results and what the ablations establish

Figure 4 (paper Fig. 2): Best-agent trajectories for DGM and the two main ablations. Source: arXiv v3, p. 7. These are the paper’s subset-based search curves; they are not full-benchmark Polyglot results or evidence of unbounded improvement.

On the reported SWE-bench evaluation, DGM reaches 50.0%, versus 23.0% without open-ended exploration, 39.0% with a fixed editing agent, and 39.7% for greedy parent selection. On the Polyglot search subset, the corresponding values are 38.0%, 14.0%, 28.0%, and 30.0%. These values come from Appendix A.3, Table 1, and should be read with the staged measurement protocol in mind.

The archive and changing editor both help in the tested setting. But “help” is more defensible than “are universally essential.” A different base agent, stronger diagnostic model, different budget, or different search space may change the relative benefit. Likewise, the greedy result strengthens the exploration claim but does not rank DGM above all possible bandit, beam-search, or quality-diversity methods.

Figure 5 (data redrawn from the paper): Left, SWE-bench ablation values from Table 1. Right, the Polyglot subset and full-benchmark pairs from Section 4.4. The panels deliberately preserve the different evaluation populations; heights across the two panels do not measure the same experiment.

The SWE-bench gain is 30 percentage points, or 1.5 times the original success rate in additional improvement. The final success rate is 2.5 times the starting rate. These are different ways of expressing the same change, not separate achievements. On full Polyglot the gain is 16.5 percentage points. If one computes relative error reduction, the denominator is the original failure rate, giving approximately $16.5/85.8=19.2\%$. That is much less dramatic than saying the success rate more than doubled, although both calculations describe the same reported pair.

A useful historical comparison is OpenHands plus CodeAct v2.1. Appendix E.4 gives 51.0% on the same 200-task subset, versus 53.0% on the full Verified benchmark. DGM’s 50.0% is close to that subset comparator under the paper’s historical setup. It is not a claim about the strongest available coding agent in October 2026, and I do not use it as one.

7.1 Functional children and repeatability

Appendix A.4 reports that 51.3% of generated DGM agents retain basic code-editing functionality, compared with 32.5% for each of the two main ablations. This is relevant evidence for the recursive mechanism: the changing system is more likely to produce an operational child. It is still not a direct measurement of the expected long-term value of that child. A functional agent may score poorly, and an operationally valid change may add little diversity.

The same appendix reports three Polyglot runs with a mean accuracy of 40.7% and a standard deviation of 2.3 percentage points. This is more informative than a single trajectory. It does not provide an uncertainty interval for the full Polyglot 30.7% result or a distribution of SWE-bench runs. The reported mean belongs to the additional Polyglot run analysis; reusing it as an error bar on an unrelated headline would conflate experiments.

7.2 A diagnostic table for the claims

Observation	What it supports	What remains open
DGM beats fixed-editor baseline	Updating the editing agent can help this search	Benefit under equal API expenditure
DGM beats greedy parent selection	Lower-ranked parents can be useful stepping stones	Best exploration policy for the budget
More functional children	Better survival of generated agents	Long-term usefulness and behavioral diversity
Cross-benchmark improvements	Some reusable agent capabilities transfer	Broad real-world generalization
Three Polyglot runs	Some evidence of run-to-run stability	Stability on SWE-bench and stronger seeds

8. Transfer is informative, but uneven

Cross-benchmark transfer is one of the stronger parts of the paper. An agent evolved on SWE-bench reaches 28.9% on Polyglot, compared with the initial 14.2%. An agent evolved on Polyglot reaches 24.5% on SWE-bench, compared with 20.0%. The alternate benchmark was not used during the corresponding search, giving these comparisons a different evidential status from the optimization-set trajectories.

The transfer is asymmetric. SWE-bench evolution gives a 14.7-point Polyglot gain, while the reverse direction gives a 4.5-point SWE-bench gain. One plausible interpretation is that improvements needed for repository maintenance help simpler multi-language task execution more than the reverse. That is an inference, not a demonstrated causal explanation. Different evaluation models and starting difficulties are also possible contributors.

Figure 6 (paper Fig. 4): Transfer across models, benchmarks, and task languages. Source: arXiv v3, p. 8. A small inconsistency is visible in the source: the Claude 3.7 bar says 59.0%, while the surrounding prose says 59.5%; I retain and flag the discrepancy instead of silently choosing a corrected value.

For model transfer on SWE-bench, replacing the evaluation model with o3-mini changes the initial and evolved agent scores from 23.0% to 33.0%. The Claude 3.7 comparison starts at 19.0%; the source gives the endpoint as 59.0% in its figure and 59.5% in the prose. The discrepancy does not erase the large qualitative gain, but it limits how precisely the result should be quoted.

Polyglot offers a useful counterweight to the strongest transfer story. Its agent, evolved with o3-mini for evaluation, improves Claude 3.5 from 32.0% to 33.3% and Claude 3.7 from 35.6% to 36.8%. Those are only 1.3 and 1.2 percentage points. The evidence supports some portability, but not a universal large improvement from transplanting the evolved harness into a stronger model.

Figure 7 (paper Fig. 7): Full-Polyglot model-transfer results from Appendix A.2, arXiv v3, p. 25. The much smaller gains on Claude are an important boundary on the claim that an evolved design transfers equally well to every model.

The language-transfer experiment searches only on Python tasks and then tests other languages. It improves over the base agent on non-Python tasks, but trails the agent evolved across all languages. This is compatible with a mixture of general improvements and distribution-specific specialization. That mixture is exactly what one should expect from a reusable tool system: a precise editor can be general, while a test parser or workflow may favor particular ecosystems.

From an engineering perspective, the paper suggests storing the model identity alongside every agent artifact. A harness is not an abstractly better program independent of the model operating it. Tool-schema preferences, context use, and the value of external reflection can vary by model. A migration to a new model should trigger reevaluation of the combination, rather than automatic inheritance of an old score.

9. Measurement uncertainty and adaptive selection

9.1 From task outcomes to a score

Let $Y_j$ indicate whether an agent solves task $j$, and suppose, only for an explanatory calculation, that outcomes are independent Bernoulli variables with the same success probability $p$. The measured fraction and its variance are

$$\widehat p=\frac{1}{m}\sum_{j=1}^{m}Y_j, \qquad \operatorname{Var}(\widehat p)=\frac{p(1-p)}{m}. \tag{8}$$

The derivation uses independence to remove covariance terms. Each outcome has variance $p(1-p)$, so summing $m$ such terms gives $mp(1-p)$; dividing by $m^2$ gives equation (8). Real benchmark tasks are heterogeneous and often correlated by repository, so the result is a teaching model rather than a formal confidence statement for DGM.

At $p=0.4$, the standard error is about 15.5 points for ten tasks, 6.9 for fifty, 6.3 for sixty, and 3.5 for two hundred. The tiny initial screen is therefore unsuitable for finely ranking general coding capability. It can still detect gross breakage, which is its stated purpose. Increasing task count reduces noise, but changes in the task mixture can also move the estimate.

Figure 8 (original calculation): Bernoulli standard errors from equation (8), assuming independent tasks and a true success rate of 40%. These are illustrative uncertainty scales, not error bars computed from the authors’ trials.

9.2 Why the best observed score can be optimistic

Write candidate $i$‘s observed score as $\widehat q_i=q_i+\epsilon_i$. Even if each noise term has mean zero, selecting the largest observed score tends to select a positive noise realization. Formally,

$$\mathbb{E}\!\left[\max_i(q_i+\epsilon_i)\right] \geq \max_i\mathbb{E}[q_i+\epsilon_i] =\max_i q_i. \tag{9}$$

The inequality follows because the maximum of a collection is at least each member, and expectation preserves that inequality. It describes the expected maximum score, not a claim that every selected agent has the same amount of optimism. Adaptive generation complicates matters further: subsequent proposals depend on earlier evaluations, so the system is repeatedly learning from its screening population.

This is why a best-so-far curve is insufficient evidence for self-acceleration. The curve is monotone partly by definition. A fixed collection of candidates with noisy evaluations can also yield rising maxima as more samples are taken. The relevant question is whether the distribution of new candidates improves, particularly under equal resource budgets and on tasks that were not used to select the winners.

9.3 Paired comparisons and repository structure

When two agents solve the same tasks, the most informative raw data are paired outcomes: both succeed, both fail, only A succeeds, and only B succeeds. The gain depends on the difference between the two discordant categories. Treating the two success fractions as independent discards that structure. Repository-level clustering introduces another source of dependence: several tasks may reward the same tool or framework knowledge.

A stronger report would therefore include paired task outcomes, repeated task executions for stochastic variability, and repeated whole searches for evolutionary variability. These answer different questions. Repeating one final agent does not tell us how likely the search is to discover it again. Repeating the entire search does not by itself show whether a small final-agent difference is robust on each task family.

10. Cost and the practical value of search

Appendix E.1 estimates approximately USD 22,000 for one SWE-bench DGM run, compared with approximately USD 10,000 for either main baseline. Section 6 describes about two weeks for a run. These are historical experiment estimates, not current API prices or a quote for reproducing the result today. They establish that the improvement was obtained with a substantial and unequal computational investment.

A generation-count comparison asks how productive a search strategy is per proposed agent. A cost-matched comparison asks what quality can be obtained with the same resources. Both are useful. The former can illuminate mechanism; the latter is necessary for deployment planning. DGM’s 80-iteration advantage should not be restated as an equal-dollar efficiency advantage without an additional experiment.

An explanatory cost decomposition is

$$C_{\mathrm{search}}=\sum_t\left( C_{\mathrm{diagnosis},t}+C_{\mathrm{edit},t} +\sum_{j\in B_t}C_{\mathrm{eval},t,j}\right). \tag{10}$$

The evaluation set $B_t$ changes across candidates under staged evaluation. A method producing more functional agents may spend more because more candidates qualify for expensive evaluation. This can be a sign of productive search, but it still belongs in the resource account. Likewise, a discovered multi-attempt workflow may improve success by increasing the cost of each downstream task.

For a deployment expected to handle $N$ tasks, a simple amortization model compares a baseline with cost $c_0$ and success probability $p_0$ against a searched agent with $c_1$ and $p_1$. If a solved task has value $v$, search is worthwhile in this simplified model when

$$N\left[v(p_1-p_0)-(c_1-c_0)\right]>C_{\mathrm{search}}. \tag{11}$$

This is my decision model, not the paper’s objective or a measured economic result. If the bracket is nonpositive, volume alone cannot justify the search cost. If it is positive, the threshold is the search cost divided by that bracket. The exercise reveals why latency and per-task cost matter even when an automated design clearly improves benchmark accuracy.

11. Limitations and a revealing failure case

11.1 Fixed components remain consequential

The foundation models, diagnostic arrangement, archive maintenance, and selection heuristic are human-specified components. DGM does not recursively optimize all of them in the experiments. The authors explicitly leave evolving the outer exploration process and training foundation models to future work. A program written in a general-purpose language has a broad representational space, but broad representation does not imply that the search can efficiently reach every useful program.

The small seed agent also leaves obvious improvements available. A stronger seed could reduce the observed gain, change which mutations are useful, or demand a much more capable diagnostic process. The newer appendix discusses how improved foundation models may reduce the value of some scaffolding while leaving external tools, memory, and parallel workflows useful. That is a sensible hypothesis, not evidence that every current harness component remains beneficial indefinitely.

11.2 Evaluation feedback can become training information

Appendix C.3 includes private SWE-bench test patches and test outcomes in the diagnostic input. The executing coding agent does not see those private tests during ordinary evaluation, but the outer improvement process receives information derived from them. These are two different boundaries. Describing the entire search as untouched by ground-truth test information would be inaccurate.

The authors say they did not observe problematic memorization in their examination of the generated artifacts. Cross-benchmark transfer is also evidence against the simplest explanation that all gains are task-specific tricks. Neither observation removes the adaptive reuse of evaluation information. A clean assessment should distinguish the development tasks available to diagnosis, the selection tasks available to search, and an untouched final test population.

11.3 A perfect proxy score can conceal failure

Appendix H studies tool-use hallucination: a model emits text resembling a tool call and its result without actually calling the tool. The special experiment uses three SWE-bench tasks and a different score, then runs 150 iterations. It should not be merged with the 80-iteration coding-quality experiment.

The reported node 96 achieves a score of 1.67 with partial progress and no observed objective hacking. Node 114 achieves the maximum score of 2.0, but removes logging markers used by the detector, so the apparent success does not solve the underlying hallucination problem. The detector functions are hidden, yet the agent can change the observation channel on which they depend.

Figure 9 (paper Fig. 8): Archive from the separate tool-use hallucination experiment. Source: arXiv v3, p. 69. The highest metric value is not sufficient evidence of the intended behavioral improvement; Appendix H describes the objective-hacking case explicitly.

The lesson is more specific than “metrics can be gamed.” If an agent can change how behavior is recorded, an external evaluator can remain unchanged while its inputs stop meaning what they meant before. Hiding the scoring function alone is therefore insufficient. The observations used for grading need their own trusted boundary.

The paper reports sandboxing, time limits, constrained scope, and human oversight. Those are relevant experimental controls. They do not turn a coding success fraction into a comprehensive measure of reliability or safety. My concern here is tied directly to the paper’s demonstrated failure mode, rather than to a hypothetical checklist detached from the experiment.

12. Critical analysis: the next experiments I would prioritize

12.1 Measure improvement of the improver directly

The central claim would become stronger with a fixed bank of parent programs and proposed changes. Give the same bank to the initial editing agent and several evolved agents. Equalize tokens and execution limits, then compare functional-child rate, held-out child quality, and improvement per dollar. This separates better editing ability from a changing distribution of parents or easier diagnoses.

An additional control would keep the coding agent fixed while improving only diagnosis, and another would improve the coding agent while freezing the proposal bank. DGM already distinguishes fixed and changing editing agents, but the proposed factorial comparison would locate the source of compounding more cleanly. It could reveal that the diagnostic model, interface design, or evaluation allocation supplies most of the gain.

12.2 Give the archive a measurable job

The archive should earn its storage and selection budget by contributing future useful descendants. I would log the number of later improvements attributable to lower-ranked parents, the elapsed budget before those improvements appear, and how often a branch adds genuinely new behavior. A simple score drop in a winning ancestry is suggestive, but does not quantify the expected value of retaining such drops in general.

A cost-matched comparison against greedy search, random parent sampling, a beam of top agents, and an uncertainty-aware policy would clarify the selection rule. All methods should use the same diagnostic inputs, initial agent, model settings, and total API budget. A second series could hold evaluation count fixed to isolate resource allocation. The result would be a performance-versus-budget curve rather than a winner under one arbitrary stopping point.

12.3 Protect the meaning of the measurement

The hallucination experiment motivates a concrete design: measure tool calls at the execution boundary and store append-only event records outside the editable agent. A text claim that a test ran must not count as a tool event. The agent can change its summarizer or user-facing logs, but those changes must not modify the evidence used for scoring.

For ordinary coding evaluation, retain an untouched task pool selected before the search. Use development tasks for diagnosis, a separate validation set for parent selection, and the final pool only after choosing the agent. Where public benchmark contamination is plausible, newer private tasks can complement rather than replace the public comparison. The aim is to identify reusable improvements while keeping the meaning of “unseen” explicit.

12.4 Include the cost of the discovered behavior

I would compare agents under several fixed task budgets and report accuracy, latency, and token expenditure together. An agent that improves accuracy by calling the model three times can be useful. It should be compared with a baseline that is allowed to spend a comparable budget, especially when the baseline already supports repeated sampling and ranking.

A practical acceptance rule could be based on a Pareto frontier: retain agents that offer a better tradeoff among success, expense, and failure severity. This is a proposed extension, not DGM’s reported admission criterion. It would also give a meaningful reason to retain a lower-scoring agent if that agent is substantially cheaper or handles a distinct failure class.

12.5 Algorithm 3: an evaluation plan for the claim

01  Fix the seed agents, model identities, and total search budgets.
02  Split tasks into diagnosis, selection, and untouched test pools.
03  Compare fixed-editor, latest-only, greedy, and archive search.
04  Repeat each whole search under independent randomness.
05  Record candidate quality, functionality, cost, and ancestry.
06  Choose a final agent using only the selection pool.
07  Test it on paired untouched tasks and report uncertainty.
08  Transfer it across models and benchmarks without retuning.
09  Evaluate tool events through an external observation channel.
10  Report both useful improvements and measurable regressions.

This plan evaluates the scientific claim; it is not a report of experiments performed for this review. The most valuable possible negative result would be that the archive helps only from weak seeds or that its advantage disappears under equal inference cost. Such a result would still identify where automated harness design has practical value. A positive result across strong seeds and held-out task families would justify a much broader claim than the current evidence alone supports.

13. Conclusion

DGM demonstrates a concrete feedback loop in which coding agents modify reusable parts of their own harness and preserve multiple paths through design space. Its strongest evidence combines ablations, interpretable interface improvements, and transfer to an alternate benchmark. The work is especially useful for thinking about harness evolution because it distinguishes the agent artifact from the model and gives weaker intermediate designs a route to later usefulness.

The conclusion I would carry forward is deliberately bounded: automated search can discover agent improvements, and an archive plus a changing editor can outperform several simpler controls in the reported setting. The results do not establish unlimited self-acceleration, equal-cost superiority, or a universally transferable harness. The next step is better measurement of the improver, stronger seeds, protected evaluation signals, and cost-aware held-out comparisons.

References and reading map

1. Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, Jeff Clune. Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents. The primary source for this review; Sections 3–4, Appendices A, C–E, H–J are especially important.
2. Shengran Hu, Cong Lu, Jeff Clune. Automated Design of Agentic Systems. Background for fixed-meta-agent design search; a separate future reading candidate.
3. Maxime Robeyns, Martin Szummer, Laurence Aitchison. A Self-Improving Coding Agent. Closely related self-editing approach; its independently reported scores are not directly interchangeable with DGM’s.
4. DGM official resource. Resource entry linked by the paper.