Skill Issue: What It Really Takes to Optimize Repository Knowledge for Coding Agents

Review date: 2026-09-14
Author: Zhongzhu Zhou
Paper reviewed: Skill Issue: Lessons from Optimizing Repository SKILLs for Coding Agents
Paper authors: Mykhailo Kozyrev, Andrei Kozyrev, Anton Podkopaev
arXiv: 2609.12742v1
Venue/Status: Preprint, September 2026

1. Why this paper matters

A repository skill is deceptively simple: it is a Markdown file loaded before a coding task. It may describe build commands, module boundaries, local conventions, and failure-prone workflows. Because it is plain text, a team can review and version it like source code.

The difficult question is not whether such a file can contain useful advice. It is whether an optimizer can discover advice that causes a fixed coding agent to perform better on future repository tasks.

The paper studies that question with an unusually honest instrument. It mines real merged pull requests, reverses their implementation at one frozen repository state, and compares a candidate skill against an empty seed on the same task. GEPA improves the mean paired score by 4.9 percentage points; SkillOpt improves it by only 0.1 points. Neither effect is statistically distinguishable from rerun variance at the available sample size.

That negative-looking result is valuable. It separates three objects that are often conflated:

• a document that reads well;
• a benchmark score that moves;
• causal evidence that the document moved the score.

My central reading is: this is less a paper about prompt optimization than a paper about measurement engineering for agents.

2. Prerequisites

2.1 Coding-agent execution model

A coding agent repeatedly observes repository state, reasons, invokes tools, edits files, and runs tests. Let task $x$ include an issue statement, initial tree, and hidden verifier. A harness $H$ combines model $M$, tools $T$, limits $L$, and optional skill $s$:

$$\tau sim H(M,T,L;,x,s)$$

Here $\tau$ is a stochastic trajectory. Even with every visible input fixed, sampling, tool timing, and recovery choices make two trajectories differ. Therefore a score difference between two single rollouts is not automatically a skill effect.

2.2 Repository knowledge versus generic advice

Useful repository knowledge is local and falsifiable: which Gradle task exercises a module, which source-set may depend on which other source-set, or which annotation registers a tool. Generic advice such as “read the README” is usually harmless but consumes context without differentiating the repository.

A good skill should maximize action-relevant local information per token, not prose quality.

2.3 Counterfactual task construction

A merged pull request contains an observed fix. To turn it into a benchmark task, we need a counterfactual world where the fix is absent but the surrounding repository remains coherent. The agent receives the problem and must reconstruct a valid solution, while the original patch is hidden.

2.4 Train, selection, and test splits

Text optimizers can overfit just like parameter optimizers. Training tasks produce reflection evidence. A selection split accepts or rejects candidate edits. A test split is touched once at the end. With roughly 100 mined tasks, dividing three ways leaves only 20–26 held-out tasks per repository; that fact later dominates statistical power.

3. The complete system at a glance

The pipeline has two coupled loops.

The data loop turns repository history into graded tasks:

1. choose one frozen base commit;
2. stream merged pull requests;
3. split implementation and test changes;
4. reverse the implementation against the base;
5. run tests before and after reversion;
6. retain only tasks with isolated passing-to-failing tests;
7. remove history and leakage before agent execution.

The optimization loop turns agent failures into a candidate document:

1. select a skill candidate and task minibatch;
2. run the fixed coding agent;
3. compare each rollout with its empty-seed counterpart;
4. reflect on transcripts and feedback;
5. rewrite or edit the Markdown;
6. keep candidates according to GEPA or SkillOpt policy;
7. evaluate the final candidate on unseen tasks.

The isolation contract is important. The model, harness, tool set, turn budget, base commit, and container image stay fixed. Only $s$ changes. This makes the intended intervention legible:

$$Delta(x,s)=Q(\tau_{x,s},\tau_{x,\varnothing})$$

where $Q$ compares a skill rollout with the stored seed rollout on task $x$.

3.1 Why a single frozen base?

The obvious alternative is the forward SWE-bench construction: check out each pull request’s parent commit, apply its tests, then ask the agent to reproduce the implementation. This yields more tasks, but every task lives at a different historical state.

If optimization sees months of repository history, it can learn contradictory facts. A path valid in March may be gone in September. The resulting skill describes a mixture of repositories that never exists at deployment.

The frozen-base choice trades quantity for temporal consistency. It works when historical edits can still be projected onto today’s tree. It fails after broad renames, module moves, or architectural rewrites. The paper’s Tracy example retained only 5 of 203 pull requests.

3.2 What is held fixed—and what is not

The authors run Claude Code with Sonnet 4.6 in fresh Docker containers. The skill is the sole optimized parameter. However, the empty-seed rollout is stored once rather than repeatedly sampled. This saves substantial cost, but it means paired comparisons still contain one draw from each stochastic condition, not estimates of their expectations.

That design is reasonable for an expensive experiment. Its boundary is causal precision: rerun noise can dominate a few-point effect.

4. Algorithm 1: mining reverse-PR tasks

A merged pull request is not automatically a valid task. The method must reconstruct the pre-fix implementation while preserving a current, buildable repository.

Numbered pseudocode — reverse-PR mining

1. Freeze repository tree at base commit $b$ and run the full test suite.
2. Record the set $P_b$ of tests that pass at $b$.
3. For each merged pull request $r$, split its patch into implementation patch $I_r$ and test patch $U_r$.
4. Try to reverse $I_r$ at $b$ with git apply —reverse.
5. If step 4 fails, structurally undo added/deleted files.
6. For remaining modified files under the size gate, ask an LLM to reconstruct pre-change source.
7. Re-derive a real Git patch from the reconstructed tree; never trust free-form model patch syntax.
8. Run static checks for missing imports and stranded fixes.
9. Execute tests on the reverted tree.
10. Define the hidden set $F_r=P_b\setminus P_r$, where $P_r$ is the set passing after reversion.
11. Reject if $F_r$ is empty, grading tests already failed at $b$, unrelated failures spread beyond the change, or the gold patch cannot restore correctness.
12. Attach at most 30 sampled regression guards from $P_b$.
13. Hide Git history, scrub links to the original pull request, and synthesize leak-checked issue text when no linked issue exists.
14. Emit task $(b,I_r,F_r,G_r,d_r)$.

The key set difference follows directly. A grading test must have passed before damage and fail after damage:

$$F_r = \{t:t\in P_b\} \setminus \{t:t\in P_r\}$$

If $F_r=\varnothing$, the reversion may be behaviorally irrelevant or untested. Such a task gives every candidate the same signal and should not consume a rollout.

4.1 Three reversion tiers

Tier 1: exact reverse patch. This is cheap and trustworthy, but survives only limited code drift.

Tier 2: structural file undo. Added files are removed and deleted files recreated. This handles file-level changes but not semantic drift inside modified files.

Tier 3: LLM reconstruction. The model sees current source plus the forward diff and reconstructs the old implementation. This contributes most of the yield: only 25 of koog’s 119 graded tasks reverse-applied cleanly; kotest had 56 of 100 and ktor 65 of 131.

Why use an LLM? Traditional three-way merge cannot recover intent when surrounding code changed substantially. Why not accept model output directly? Because it can invent syntax or unrelated edits. Re-deriving the patch through Git constrains the artifact to an auditable tree delta.

4.2 A blind spot in dynamic validation

The dynamic gate only asks whether reversion breaks tests. An LLM can delete a still-needed import, causing compilation failure that masquerades as a useful test failure. It can also remove a call while leaving a private helper containing the answer.

On an intermediate 280-task koog pool, a static check rejected 56 tasks: 36 missing-import defects, 26 stranded fixes, and 6 with both. Thirty-six had already passed dynamic validation. This is a strong lesson: a verifier validates only the property it measures.

5. Algorithm 2: paired rollout scoring

Absolute pass rate mostly measures task easiness. If an empty skill already solves a task, every candidate gets credit despite contributing nothing. The paper instead asks whether the candidate does better than the seed.

Let $a$ and $z$ denote candidate and seed rollouts. The comparator is lexicographic.

Numbered pseudocode — paired score

1. Load stored seed rollout $z_x$ for task $x$.
2. Run the same harness on $x$ with candidate skill $s$, producing $a_{x,s}$.
3. If exactly one rollout tampers with tests or makes a contradicted success claim, mark that rollout the loser.
4. Else, if exactly one rollout passes every hidden FAIL_TO_PASS test and regression guard, mark it the winner.
5. Else compute bounded tie features: hidden-test fraction, claim honesty, diff size, and tool calls.
6. Clip tie features so tidy or cheap failure cannot outrank correctness.
7. Return $q=1$ for candidate win, $q=0$ for candidate loss, and $q=0.5$ for an exact tie.
8. Average $q$ over the split.

A compact representation is:

$$q_x(s) = \begin{cases} 1, & a_{x,s}\succ z_x,\\ \tfrac{1}{2}, & a_{x,s}\equiv z_x,\\ 0, & a_{x,s}\prec z_x. \end{cases}$$

For $n$ tasks, the reported score is:

$$S(s) = \frac{1}{n} \sum_{i=1}^{n} q_{x_i}(s)$$

The seed compared with itself has $S(\varnothing)=0.5$. Therefore the centered effect is:

$$\widehat{\Delta}(s) = S(s)-0.5 = \frac{W-L}{2n}$$

where $W$ and $L$ count candidate wins and losses; ties cancel. This derivation exposes why a 0.55 score is modest: it corresponds to only a 0.05 centered advantage, not a 55% absolute success rate.

5.1 Why lexicographic ordering?

Correctness must dominate thrift. A weighted sum such as

$$R = \alpha C - \beta D - \gamma K$$

could let a tiny, cheap, incorrect patch beat a costly correct patch if weights are poorly calibrated. Lexicographic comparison first separates cheating and correctness, then uses efficiency only as a tie-breaker.

The alternative is a learned judge. But a skill-blind judge scores the patch rather than the skill’s causal contribution, while a skill-reading judge can reward polished documentation that does not change agent behavior.

The boundary is partial correctness. If neither rollout fully passes, the tie-break layer compresses many qualitatively different failures into one bounded number. Better semantic progress metrics could provide denser signal without violating correctness priority.

5.2 Pairing reduces, but does not eliminate, variance

Pairing controls task difficulty because candidate and seed face the same $x$. Yet it does not pair random seeds or trajectories. In causal notation, the target is:

$$\Delta_x = \mathbb{E}[Y\mid x,s] - \mathbb{E}[Y\mid x,\varnothing]$$

while the experiment observes one sample from each conditional distribution. Repeated paired rollouts or common-random-number controls would reduce uncertainty, but multiply cost.

6. Algorithms 3 and 4: how the documents evolve

6.1 GEPA: whole-document reflective search

GEPA treats the Markdown as a textual parameter and the reflection model as a mutation operator.

Numbered pseudocode — GEPA adaptation

1. Initialize candidate pool with seed skill $s_0$.
2. Maintain a Pareto frontier over selection-task behavior.
3. Select parent $s_p$ from the frontier.
4. Sample a training minibatch $B$.
5. Run the target coding agent with $s_p$ on every task in $B$.
6. Give scores, trajectories, and feedback to a reflection model.
7. Ask the reflector to rewrite the complete skill, producing $s'$.
8. Re-evaluate $s'$ on $B$.
9. If $s'$ improves over $s_p$, add it to the pool and evaluate it on a larger selection set.
10. Update the Pareto frontier and repeat until the attempt budget is exhausted.
11. Return the selected frontier candidate.

Why whole-document rewriting? It can reorganize conflicting guidance and escape local edit constraints. The obvious alternative is patch-level mutation. Whole rewrites can also delete valuable details or introduce broad generic advice; selection noise then decides whether the loss survives.

6.2 SkillOpt: bounded edits with rejection memory

SkillOpt evolves one document through add, delete, and replace operations.

Numbered pseudocode — SkillOpt adaptation

1. Initialize current skill $s_0$, edit budget $e_0$, and empty rejected-edit buffer.
2. Run $s_t$ on a training batch and partition successes and failures.
3. Ask the optimizer model to propose bounded edits from trajectory evidence.
4. Remove edits similar to epoch-local rejected edits.
5. Rank remaining edits by expected utility.
6. Apply at most $e_t$ top edits to produce candidate $s'$.
7. Evaluate $s'$ on the held-out selection split.
8. Accept only if $S_{sel}(s')>S_{sel}(s_t)$; otherwise retain $s_t$ and store rejected edits.
9. Decay $e_t$ according to schedule.
10. At epoch boundary, fold durable lessons into a protected slow/meta field.
11. Repeat for three epochs and return the final accepted skill.

The edit budget acts like a textual learning rate. Early larger edits explore; later smaller edits stabilize. The rejection buffer resembles taboo search, while the protected meta field resembles momentum.

The strict-improvement gate sounds safe but is brittle under noisy, quantized selection scores. If a split has $m$ tasks, the score often moves in increments near $1/(2m)$ or $1/m$. A genuinely better edit can be rejected because one stochastic rollout flips.

6.3 Why GEPA may win here

GEPA can preserve several behaviorally distinct candidates on a Pareto frontier. SkillOpt follows a single lineage and requires immediate measured improvement. Under high noise and sparse credit, diversity is valuable.

Yet the study does not establish an optimizer-level causal conclusion. There are only three repositories, one final run per configuration, and both optimizers consume different schedules and costs. “GEPA wins this benchmark” is supported; “GEPA is generally superior” is not.

7. What the mined datasets look like

The study uses JetBrains/koog, kotest/kotest, and ktorio/ktor. They are Kotlin/JVM repositories that SWE-smith’s Python AST and pytest assumptions do not cover.

The yield is harsh:

Repository	Merged PRs examined	Graded tasks retained	Approximate yield
koog	660	119	18.0%
ktor	452	131	29.0%
kotest	capped at 100 tasks	100	not comparable

A new repository needs all of these properties:

• historical touched files still exist at the frozen base;
• the base suite is green;
• reversing a change breaks at least one previously passing test;
• failures isolate that change;
• the gold patch restores the target;
• enough survivors remain for three disjoint splits.

Tracy fails temporal applicability: package renames leave 5 of 203 PRs. http4k gets 150 reverse-applied changes from 700 PRs but falls below 100 after behavioral validation. More history is not equivalent to more signal.

7.1 Why these tasks are harder than synthetic mutations

Released SWE-smith tasks used by closely related work change a median of 4 or 7 lines and are single-file in 98–100% of cases. Reverse-PR tasks are larger:

Repository	Median changed lines	Median files	Single-file share	Empty-skill resolve
koog	54	3	23%	40%
ktor	27	2	50%	66%
kotest	16	1	73%	51%

The pooled empty-skill resolve rate is 53%. This is desirable because a benchmark saturated near 100% cannot show improvement through binary pass rate.

The obvious alternative is to deliberately weaken the harness or use a smaller model. That creates headroom, but optimizes a skill for a proxy agent rather than the production agent. The authors instead keep the intended strong harness and make tasks harder.

The boundary is selection bias. Surviving reverse-PR tasks favor changes whose files remain stable and whose behavior is well tested. They may underrepresent architecture migrations, documentation tasks, dependency upgrades, and cross-service changes—the exact work where repository guidance might matter most.

7.2 Leakage control

Using a pull request description risks revealing the implementation because it was written after the code. When a linked issue is unavailable, the pipeline rewrites the PR description into issue-style text, checks it against the gold patch, removes links back to the PR, deletes Git history, and creates a single fresh commit.

These steps reduce direct leakage. They cannot eliminate semantic hints: issue text may still name the right module or API. A stronger future audit would compare agent performance under real issue text, synthesized text, and deliberately minimal text.

8. Main experimental results

Paper Table 1 reports the final held-out paired scores and full optimization cost.

Repository	GEPA score	SkillOpt score	GEPA resolved delta	SkillOpt resolved delta
koog	0.525	0.546	+1	+2
kotest	0.554	0.519	+3	+1
ktor	0.567	0.437	+4	-2

GEPA beats the 0.5 seed reference on all three repositories. SkillOpt gains on two and regresses on ktor. Averaged as reported, GEPA adds 4.9 percentage points and SkillOpt adds 0.1.

There is a subtle interpretation issue. The score includes ties and bounded tie-breakers, while “resolved delta” counts fully solved tasks. A candidate can therefore improve paired score without adding many complete resolutions, or solve one more task while losing efficiency on several ties. Both views should be reported.

8.1 Cost and wall clock

Across six runs, total spend is $2,013.98 and wall time sums to 69.2 hours. Per repository:

Repository	GEPA spend / time	SkillOpt spend / time
koog	$368.82 / 4.7 h	$401.44 / 8.14 h
kotest	$182.48 / 4.9 h	$263.56 / 13.3 h
ktor	$312.87 / 13.0 h	$484.81 / 25.2 h

A candidate-task evaluation is a full coding-agent rollout, averaging about $0.84. GEPA receives 200 scored attempts per repository. The expensive part is not generating Markdown; it is measuring whether Markdown changes behavior.

This explains why “just collect more samples” is not operationally trivial. Doubling test precision competes with optimizer exploration for the same rollout budget.

8.2 Maintainer evaluation and live issues

A koog maintainer found both optimized documents contained hard-earned repository knowledge, especially Kotlin multiplatform source-set boundaries and dependency direction. They also found generic padding and omissions such as the important @Tool annotation.

Two open issues provide suggestive operational evidence:

Issue	Configuration	Cost	API time	Wall time
#1275	no skill	$6.06	8m21s	14m59s
#1275	GEPA	$2.48	3m35s	5m32s
#1275	SkillOpt	$2.44	3m43s	5m30s
#1354	no skill	$4.85	7m21s	20m29s
#1354	GEPA	$3.10	4m52s	7m38s
#1354	SkillOpt	$4.20	5m33s	8m24s

These six runs are not a powered experiment, but they reveal an effect binary pass/fail hides: skills may reduce search and recovery cost even when all configurations eventually pass.

9. Statistical power: why the headline cannot carry the claim

For tasks where candidate and seed disagree, the null hypothesis says either rollout is equally likely to win. If $D=W+L$ excludes ties, then under no skill effect:

$$W \sim \operatorname{Binomial}(D,0.5)$$

The one-sided exact sign-test probability is:

$$p = \Pr(X\ge W\mid D,0.5) = \sum_{k=W}^{D} {D\choose k} 2^{-D}$$

No optimizer run achieves $p<0.05$; the best reported value is $0.29$.

9.1 A worked example

Suppose only $D=20$ task pairs disagree. To reject at one-sided 5%, we need the smallest $W$ such that the binomial tail is below 0.05.

For $W=15$:

$$\Pr(X\ge 15) = \frac{ {20\choose15} + {20\choose16} + \cdots + {20\choose20} }{ 2^{20} } \approx 0.0207$$

Thus 15 of 20 discordant pairs—75%—are needed. A mild 55:45 advantage is invisible at this scale. The paper summarizes the per-repository requirement as roughly four of every five disagreements; pooled over 69 tasks, about two of three.

9.2 Minimum detectable effect intuition

For a rough normal approximation, a two-sided test of a paired win probability $p$ against $0.5$ needs:

$$n \approx \frac{ z_{1-\alpha/2}^{2} \cdot 0.25 }{ (p-0.5)^2 }$$

At $\alpha=0.05$, $z_{0.975}=1.96$. To detect $p=0.60$:

$$n \approx \frac{ 1.96^2\cdot0.25 }{ 0.10^2 } \approx 96$$

This is before adding desired power, ties, repository heterogeneity, or stochastic rollout noise. A conventional 80% power calculation is larger. Therefore 20–26 held-out tasks cannot reliably detect effects of the reported magnitude.

9.3 Why averaging repositories is not a free fix

Pooling yields 69 test tasks, but assumes comparable effects and independent task-level outcomes across repositories. Tasks share code, tests, optimizer history, and one selected document per repository. A hierarchical model would treat repositories as clusters:

$$\Delta_r \sim \mathcal{N}(\mu,\tau^2)$$

with task outcomes conditional on repository effect $\Delta_r$. With only three repositories, between-repository variance $\tau^2$ is itself weakly estimated.

The right conclusion is not “skills do nothing.” It is “this experiment does not identify a small average effect.” The maintainer review and live-issue traces remain useful qualitative and operational evidence, but they answer different questions.

10. Design choices, alternatives, and boundaries

10.1 Real merged changes versus synthetic defects

Why it works: merged changes represent coherent developer work and often span files. They avoid saturation by a strong agent.

Obvious alternative: mutate a function or assertion automatically. This is cheap and scalable.

Boundary: PR-derived tasks inherit project-specific development practice and survivor bias. Synthetic tasks offer broader controlled coverage and may be better for unit-level capabilities. A mixed benchmark would separate local repair from repository-level navigation.

10.2 Reverse construction versus forward construction

Why it works: every task and every optimized claim refers to one deployable repository state.

Obvious alternative: use each PR parent commit, maximizing patch applicability.

Boundary: reverse construction loses tasks after code drift and may require an LLM reverter. Forward construction remains appropriate when the artifact being optimized is version-conditioned or retrieved per commit.

10.3 Empty seed versus handcrafted seed

Why it works: the empty seed measures the total contribution of discovered advice.

Obvious alternative: start from an expert-authored guide.

Boundary: an empty file makes optimizers spend capacity rediscovering generic harness instructions. In production, the relevant estimand may be marginal improvement over an existing AGENTS.md, not improvement over nothing.

10.4 Same production agent versus cheap proxy

Why it works: optimization targets the agent that will actually consume the skill; there is no transfer assumption.

Obvious alternative: search cheaply with a smaller model, then validate with the production model.

Boundary: direct optimization is expensive. A multi-fidelity strategy could screen candidates with a proxy and reserve production rollouts for uncertain comparisons, but only after measuring rank correlation between agents.

10.5 Tests as specification

Why it works: tests provide deterministic, executable grading and recover targets even when a PR adds no tests.

Obvious alternative: compare with the gold patch or ask an LLM judge.

Boundary: tests miss maintainability, API quality, architecture, security, and latent regressions. Gold-patch similarity penalizes equivalent solutions; judges introduce calibration and preference bias. No single verifier is sufficient.

10.6 Pairwise objective versus absolute reward

Why it works: pairing subtracts task easiness and asks a causal question about the document.

Obvious alternative: optimize full resolve rate.

Boundary: stored one-shot baselines remain noisy. Sequential repeated baselines, bootstrap confidence intervals, and outcome decomposition would better characterize uncertainty.

10.7 One document versus retrieval memory

Why it works: a Markdown artifact is reviewable, versioned, portable, and requires no runtime service.

Obvious alternative: embedding retrieval over trajectories or a self-updating memory database.

Boundary: one document has a context budget and becomes stale. Large monorepos may need hierarchical skills with explicit routing. Static text is strongest for stable invariants, not fast-changing facts.

10.8 Strict acceptance versus probabilistic acceptance

Why it works: SkillOpt’s gate prevents measured regressions and keeps evolution interpretable.

Obvious alternative: accept uncertain changes or maintain multiple candidates.

Boundary: strict improvement on tiny noisy splits freezes useful edits. Bayesian or racing-based acceptance could allocate repeats only near the decision boundary.

11. Reproducibility blueprint

The paper is unusually specific about model, repositories, task counts, attempt budgets, cost, and failure modes. A faithful reproduction still requires careful controls.

11.1 Data construction checklist

1. Pin repository URL, frozen commit, PR cutoff, and API pagination order.
2. Persist raw PR metadata and implementation/test patch classification.
3. Record which reversion tier handled each file.
4. Store the reconstructed tree and Git-derived patch.
5. Run base tests before every mining batch or prove image immutability.
6. Log rejection reason at every gate.
7. Version the static checker for missing imports and stranded fixes.
8. Preserve synthesized issue prompts and leakage-audit outputs.
9. Record hidden FAIL_TO_PASS tests and sampled guards privately.
10. Publish aggregate attrition without exposing benchmark answers.

11.2 Agent-control checklist

• pin exact model snapshot, harness version, system prompt, tools, timeout, and turn cap;
• use identical container images and resource limits;
• remove Git history and network access consistently;
• record temperature and decoding controls;
• capture tool calls, diffs, test logs, cost, API latency, and wall time;
• repeat a subset to estimate within-condition variance;
• blind maintainers to optimizer identity when rating documents.

11.3 Optimizer-control checklist

GEPA and SkillOpt should receive the same train/selection/test split, comparable rollout budgets, and identical feedback fields. Report candidate count and effective evaluations, not only nominal iterations. Save every candidate so selection instability can be audited.

A useful result record per attempt is:

Field	Purpose
repository / base SHA	state identity
task id / PR id	provenance
skill hash	treatment identity
baseline rollout hash	pair identity
hidden test outcome	primary correctness
tampering / honesty flags	guardrail
diff size / tool calls	tie-break evidence
model cost / latency	operational metric
random seed if supported	variance analysis

11.4 Sanity tests before a costly run

First, the gold patch must score as fully correct. Second, seed-versus-itself must return exactly 0.5. Third, a deliberately harmful skill should lose. Fourth, re-running the empty skill on a small subset should reveal noise magnitude. Fifth, a skill containing an intentionally unique repository fact should be observed in agent behavior on a targeted task.

Without these checks, a flat optimization curve is ambiguous: the optimizer may be weak, the metric insensitive, the skill unread, or the dataset invalid.

12. Broader technical lessons

12.1 Optimize information, not documentation aesthetics

A document can score highly for factuality and policy quality yet distract the agent. The appendix reports cases where long maps consumed a 14-turn budget and produced no patch. The correct objective includes downstream action.

12.2 Measurement headroom is necessary but insufficient

Lowering empty-skill solve rate from saturation to 53% creates room. It does not create statistical power. Headroom concerns outcome range; power concerns sample size, variance, and effect magnitude.

12.3 Repository invariants are the durable target

The maintainer valued source-set boundaries and dependency direction—rules stable across tasks. Optimizers should explicitly distinguish invariant guidance from task-specific facts. A candidate sentence could carry provenance, confidence, and last-verified commit.

12.4 Efficiency may be the first measurable benefit

On two live issues, skills sharply reduce time and cost. A staged endpoint could order outcomes as: correctness, regression safety, time-to-first-valid-test, total cost, and patch simplicity. This would detect search guidance before it changes final pass rate.

13. Limitations reported by the authors

The pipeline applies only when repository history yields enough valid tasks. Three splits require a retained pool near or above 100, but that threshold is logistical, not a proof of adequate power.

Fast-moving repositories fail reverse applicability. Weak, flaky, or already-red test suites fail behavioral validation. Changes without isolated executable effects disappear. The method therefore cannot be assumed to cover a new repository.

The authors also emphasize that optimized skills may state stale or overgeneral claims with an authoritative tone. Every synthesized skill should be reviewed like a pull request. The paired score certifies neither factual accuracy nor safe guidance.

Finally, the maintainer study covers one maintainer on one repository, and the live-issue comparison covers two issues. These observations are illuminating but not generalizable effect estimates.

14. Critical Analysis

14.1 Paper-specific weaknesses and flaws

W1 — one stored seed rollout is too fragile for a variance paper. The main claim is that rerun variance masks skill effects, yet each candidate is paired against one stored seed trajectory. That controls task identity but not baseline sampling error. Repeating the seed on even a stratified subset would quantify how much uncertainty comes from baseline luck.

W2 — optimizer comparison is confounded by protocol and budget. GEPA and SkillOpt differ in mutation granularity, candidate population, acceptance policy, attempts, spend, and wall time. The study fairly reports outcomes but cannot isolate which design causes the gap.

W3 — the paired score is insufficiently decomposed in the headline. A 0.554 score blends full wins, ties, and bounded secondary criteria. Readers need a per-repository contingency table: both pass, only skill passes, only seed passes, neither passes, tampering, and tie-break-only wins.

W4 — task attrition changes the target population. The valid pool excludes large refactors and poorly tested changes. The paper discusses yield, but not a semantic taxonomy of discarded versus retained work. The optimized skill may be best for stable, test-rich leaf changes rather than everyday repository work broadly.

W5 — human evaluation lacks blinding and replication. One koog maintainer reads documents whose provenance may be recognizable. There is no rubric agreement, multiple-maintainer variance, or comparison with a human-authored repository guide.

14.2 Limitations that are understated or omitted

U1 — context displacement is not directly measured. A skill consumes tokens and attention. The experiment records costs and tool calls but does not isolate whether failures arise because incorrect advice misleads or because useful task context is displaced.

U2 — contamination is broader than hidden Git history. A frontier coding model may know public repositories, APIs, or merged changes from training or browsing. Removing .git prevents direct lookup inside the container, not model-memory contamination.

U3 — dependence among tasks weakens nominal sample size. Multiple PRs touch shared modules, conventions, and tests. Treating each task pair as independent makes sign-test power look better than it may be.

U4 — model-version durability is unknown. A skill optimized for Sonnet 4.6 may become redundant, harmful, or differently interpreted by another model or harness. Cross-model transfer is central to the value of a repository artifact.

U5 — security risk deserves a dedicated protocol. Repository text is executable influence over an agent. Automated optimization could preserve prompt injection, unsafe commands, or test-specific shortcuts unless content is statically and manually reviewed.

14.3 Concrete improvements

I1 — use repeated, adaptive paired evaluation. Run one pair for all candidates, then allocate additional pairs when confidence intervals overlap. Sequential probability ratio tests or Bayesian racing can spend repeats only where selection is uncertain.

I2 — publish outcome decomposition and hierarchical intervals. Report discordant wins/losses, full-resolution changes, tie-break contributions, repository-cluster bootstrap intervals, and sensitivity to each score component.

I3 — build a mixed task suite. Combine reverse PRs, forward historical tasks, controlled synthetic mutations, architecture questions, and live issues. Tag each by file count, subsystem, change type, and test strength. This reveals which skill knowledge transfers.

I4 — compare against strong human baselines. Evaluate empty skill, README/CONTRIBUTING, maintainer-authored skill, retrieval memory, GEPA, and SkillOpt under equal context and rollout budgets. The practical question is whether optimization beats documentation a maintainer can write in an afternoon.

I5 — make skill claims typed and auditable. Each local fact should include scope, evidence path, last-verified commit, and confidence. CI can invalidate claims when files or commands disappear.

I6 — optimize a constrained utility. Put correctness and safety behind hard gates, then minimize expected cost among passing candidates:

$$s^* = \arg\min_s \mathbb{E}[C(s)] \quad \text{subject to} \quad \Pr(\text{correct}\mid s) \ge \Pr(\text{correct}\mid\varnothing)-\epsilon$$

This formulation captures the live-issue observation without allowing a cheaper wrong solution to win.

I7 — evaluate model and harness transfer. Cross the learned document with at least two models and two agent harnesses. Stable gains indicate repository knowledge; gains confined to one harness indicate prompt coupling.

14.4 Overall assessment

The paper’s strongest contribution is epistemic discipline. It refuses to turn a 4.9-point movement into a significance claim, traces why a repository cannot cheaply supply enough independent tasks, and checks artifacts with a maintainer.

Its weakest point is that the evaluation framework is more convincing than the optimizer comparison. That is still a meaningful result: in agent systems, building a valid intervention and metric can be harder than proposing another optimization loop.

15. Practical deployment guidance

For a team considering repository skills, I would not begin with autonomous optimization.

1. Ask maintainers for stable invariants and common traps.
2. Keep the skill short enough that every sentence earns context.
3. Attach executable evidence to commands and module claims.
4. Validate it on real historical changes at one current base.
5. Compare against no skill and existing documentation.
6. Record cost and time, not only pass/fail.
7. Require code-owner review before merge.
8. Revalidate after structural repository changes.
9. Archive skill versions with benchmark results.
10. Treat model upgrades as distribution shifts requiring a new check.

Optimization becomes attractive when the team already has a trustworthy task generator, enough non-saturated tasks, and a budget for repeated rollouts. Otherwise, the optimizer mostly amplifies weaknesses in the benchmark.

16. Conclusion

Skill Issue asks the right counterfactual: does the same coding agent perform better on the same repository task when this document is loaded?

Reverse-PR mining at a single frozen base produces harder, temporally coherent tasks. Paired scoring removes much task-easiness bias. GEPA finds documents with a +4.9-point average movement; SkillOpt is nearly flat. Small test sets and stochastic rollouts prevent those changes from being separated from chance.

The most actionable finding is qualitative. Maintainers recognize valuable local knowledge in the generated files, and two live issues become cheaper and faster with either skill. That suggests repository skills may first improve search efficiency and operational cost, before a binary benchmark can show more solved tasks.

The paper therefore supports a cautious engineering conclusion: versioned repository knowledge is promising, but optimizing it requires a benchmark, a causal comparator, statistical power, and human review. A polished Markdown file is not evidence. A moved pass rate is not necessarily evidence. The contribution lies in making those distinctions measurable.