July 27, 2026 EN #LLM Agent #Reasoning

The Regression Tax: Why Adding Skills to an LLM Agent Also Breaks Tasks It Used to Solve

Review date: 2026-07-27 Review author: Zhongzhu Zhou Paper reviewed: The Regression Tax: Decomposing Why Skills Help — and Hurt — LLM Agents Paper authors: Darshan Tank, Baran Nama (Sentient Labs) arXiv: 2607.22520v1, 2026-07-24 Venue/status: arXiv preprint (cs.AI)

0. Why this paper is worth a careful read

Every “agent skills” system I’ve looked at in the last year — Anthropic’s skill-creator, OpenAI’s Codex skill guide, and a growing pile of research systems like Trace2Skill, EvoSkill, SkillOpt — reports the same kind of number: average task success went up by X points after we added skill Y. That is the entire evaluation story. It is also, this paper argues, a badly incomplete one, and once you see why, it’s hard to unsee it in every other “we added a scaffold and accuracy went up” paper you’ll read this year.

Here’s the problem in one sentence: an aggregate pass-rate delta cannot tell you whether a skill helped tasks that were already failing, or broke tasks that used to pass — and it turns out both things are always happening simultaneously, at a scale large enough to matter. The paper runs the same agent, the same tasks, the same everything, except toggling a skill library on or off, across 5,832 paired runs. It finds that skill libraries un-fail 553 task-conditions and re-fail 324 of them (as regressions against a different set of previously-passing tasks) — so more than half of every library’s gross gains are silently cancelled by newly introduced failures. A library that looks like a clean +5-point win on the leaderboard might really be “+15 solved, −10 broken,” and that decomposition is invisible unless you look at paired, per-task transitions rather than the marginal pass rate.

What elevates this from “yet another negative result about prompt engineering” to something worth a careful read is that the authors don’t stop at “skills sometimes hurt.” They build a labeling scheme — grounded in actually reading paired execution traces, not just eyeballing numbers — that assigns why each regression happened to one of three mechanisms: (1) skill-description osmosis, where a skill changes behavior just by sitting in the system prompt, even when its body is never read; (2) grounding displacement, where an invoked skill’s procedure overrides a correct reading of the input; and (3) verification displacement, where an invoked skill suppresses a check the bare agent would otherwise have run. Each of these is falsifiable and traceable in a transcript, which is a rare property for “agent behavior” claims in this literature. The paper is also honest about its own statistical fragility — after Bonferroni correction across 18 comparisons, only 3 of the original 5 “significant” library effects survive — which is the kind of self-skepticism that makes the surviving claims more, not less, credible.

If you build, evaluate, or ship “agent skills,” “tool guides,” “system prompts as playbooks,” or any of the growing zoo of in-context procedural scaffolds, this paper gives you a vocabulary and a measurement protocol for a failure mode you are almost certainly already causing and not measuring.

1. Prerequisites

1.1 What an “agent skill” actually is

In the current wave of agent harnesses (Claude Code, Codex/OpenCode, and similar CLI-driven coding/office agents), a skill is a small, structured, natural-language package that gets loaded into the agent’s context to steer how it performs a class of tasks. Concretely it has:

a name and short description (usually one or two sentences: “use this skill when the task involves X”), which sits directly in the system prompt or a tool-listing block so the agent can decide whether to invoke it;
a body: a longer natural-language document with step-by-step procedural guidance, sometimes bundled with helper code or CLI snippets, which is only pulled into context when the agent decides to invoke/read the skill.

This two-tier structure — always-resident description, conditionally-resident body — is the load-bearing design detail of the whole paper. Most existing evaluation and defense methods (masking a skill, retrieval-gating a skill, computing a regression budget for a skill) only reason about the body: they ask “if this skill’s body were absent, would the outcome change?” They cannot see an effect that comes purely from the description being visible in every turn’s context, because they never run the counterfactual where the description is present but the body is never invoked. That gap is exactly where the paper’s first mechanism (osmosis) lives.

1.2 The office-automation benchmarks used

The paper deliberately picks two benchmarks that are neither toy nor adversarial — they’re utility tasks any office worker would recognize, and importantly the type of error each benchmark makes possible differs, which lets the paper decompose mechanisms cleanly:

OfficeQA-Pro: question-answering over real U.S. Treasury financial PDFs — extract a figure or definition, compare values across years/reports, interpret a table, compute a ratio or percentage change, combine facts from multiple documents. Answers are single numbers graded against ground truth within about 1% tolerance. Because the answer is one toleranced scalar, a regression here is unambiguous: either the number is right or it isn’t, and the trace almost always shows exactly which input the agent misread or which arithmetic step went wrong.
SpreadsheetBench: real Excel-forum spreadsheet-editing tasks — locate/filter data, write formulas, fill or update cells, sort, dedupe, work across sheets. Cell-level tasks touch a specified range; sheet-level tasks touch a broader region. Grading compares target-cell values, recalculating any formula the agent leaves in place with a grading engine that has a known blind spot: it cannot recompute modern Excel functions like AGGREGATE or structured/table references. This single grading detail turns out to matter enormously (see Section 5).

1.3 Paired comparison and McNemar’s test

The methodological backbone is simple but disciplined: run the same task, with the same model and harness, under two conditions — no skills (“none”) and with a skill library (“treatment”) — and record pass/fail for both. Because the two runs share the task, model, and harness, any difference in outcome is attributable (modulo run-to-run stochasticity, discussed later) to the presence of the skill library. This is a matched-pairs design, and the correct significance test for matched binary outcomes is McNemar’s test, which ignores tasks that agree in both conditions (both pass, or both fail) and tests only the discordant pairs — tasks that flip from pass→fail or fail→pass. If $b$ is the number of pass→fail flips (regressions) and $c$ is the number of fail→pass flips (gains), the null hypothesis is $b = c$ (the library has no net directional effect), and the exact two-sided McNemar test computes a p-value from the binomial distribution over the $b+c$ discordant pairs under $p=0.5$ . This is the correct instrument here precisely because the paper’s whole argument is that $b$ and $c$ can both be large even when $c - b$ (the net effect, i.e. the thing an aggregate pass-rate delta reports) looks modest or even flatters a library that is actually trading many correct answers for a few new ones.

1.4 Multiple-comparisons correction (Bonferroni)

With 18 independent library-vs-none comparisons (3 stacks × 2 benchmarks × 3 libraries), testing each at the nominal $\alpha = 0.05$ risks a false-positive rate far above 5% across the whole family of tests. The Bonferroni correction is the simplest fix: to hold the family-wise error rate at $\alpha$ , require each individual test to clear $\alpha / m$ where $m$ is the number of comparisons — here $0.05/18 \approx 0.0028$ . It is conservative (it can under-report true effects when tests are correlated), but it is the right first move when you don’t want to over-claim from a grid of tests run on overlapping data, and the paper uses it against itself rather than cherry-picking the nominally-significant cells.

2. Architecture / experimental-design overview

The whole study is really an experimental design more than a system architecture, so the first figure to internalize is how the four conditions and three model–harness stacks compose into the 5,832 runs.

flowchart TB
    subgraph Stacks["3 Model-Harness Stacks"]
        S1["OpenCode<br/>+ MiniMax-M2.7"]
        S2["Codex<br/>+ GPT-5.4-mini"]
        S3["Claude Code<br/>+ Claude Sonnet 4.6"]
    end
    subgraph Benchmarks["2 Benchmarks"]
        B1["OfficeQA-Pro<br/>N=94 tasks"]
        B2["SpreadsheetBench<br/>N=392 tasks"]
    end
    subgraph Conditions["4 Conditions per (stack, benchmark)"]
        C0["none<br/>(no skills)"]
        C1["anthropic library"]
        C2["openai library"]
        C3["Ours library"]
    end
    Stacks --> Benchmarks
    Benchmarks --> Conditions
    Conditions --> Runs["486 tasks x 4 conditions x 3 stacks<br/>= 5,832 paired task-condition runs"]

Figure 1 (paper Fig.1 companion, architecture): the full-factorial design. Every cell in this cross-product is run once, and every treatment condition is paired against the same none baseline within its (stack, benchmark) cell — this pairing is what makes McNemar’s test valid.

The three model–harness stacks are deliberately heterogeneous — a different agent framework (OpenCode/Codex/Claude Code) is coupled to a different underlying model (MiniMax-M2.7/GPT-5.4-mini/Claude Sonnet 4.6) in each case. This is a limitation the paper is upfront about (harness and model are confounded — you can’t ask “is this the model or the harness?”), but it buys generality: any pattern that survives across all three stacks is unlikely to be an artifact of one particular agent scaffold or one particular model’s quirks.

2.1 The always-resident description vs. conditionally-resident body

This is the architectural detail that makes the whole mechanism taxonomy possible, so it’s worth drawing out on its own:

flowchart LR
    subgraph SystemPrompt["Agent's system prompt / tool listing (every turn)"]
        D1["Skill A: name + description"]
        D2["Skill B: name + description"]
        D3["Skill C: name + description"]
    end
    subgraph ConditionalLoad["Loaded only if agent invokes the skill"]
        Body1["Skill A: full body<br/>(procedure, code, examples)"]
    end
    D1 -. "agent decides to invoke" .-> Body1
    D2 -. "never invoked on this task" .-> X["body never enters context"]

Figure 2 (architecture detail): descriptions are unconditionally present in every turn’s context; bodies are gated behind an invocation decision the agent makes at runtime. Any method that only measures “what changes if I remove/mask a skill’s body” is blind to effects produced purely by the description sitting in context — this is precisely the osmosis channel in Section 4.

3. Method: building the libraries, defining outcomes, classifying regressions

3.1 The four-stage skill-library production pipeline

To get multiple, independently-authored skill libraries from the same underlying evidence (so that library differences reflect authoring style, not different failure signals observed), the paper runs a controlled four-stage pipeline once per stack:

flowchart LR
    A["Stage 1: Baseline run<br/>agent executes with NO skills,<br/>trajectories recorded"] --> B["Stage 2: Analyst<br/>reads trajectories,<br/>extracts recurring failure signals"]
    B --> C["Stage 3: Meta-skill creator<br/>(anthropic / openai / Ours)<br/>turns signals into skills"]
    C --> D["Stage 4: Evaluation run<br/>benchmark re-run with<br/>skills in context"]

Figure 3 (pipeline): the analyst runs once per stack, so all three meta-skill creators start from the same failure signals — the only thing that varies across the three resulting libraries is how each creator turns shared signals into skill text.

Numbered pseudocode for the pipeline:

1.  baseline_trajectories = []
2.  for task in benchmark.tasks:
3.      trace = agent.run(task, skills=None)
4.      baseline_trajectories.append(trace)
5.  failure_signals = analyst.extract_recurring_causes(baseline_trajectories)
6.  for creator in [anthropic_creator, openai_creator, ours_creator]:
7.      library = []
8.      for signal in failure_signals:
9.          skill = creator.author_skill(signal)     # differs per creator, see 3.2
10.         library.append(skill)
11.     evaluation_traces = []
12.     for task in benchmark.tasks:
13.         trace = agent.run(task, skills=library)
14.         evaluation_traces.append(trace)
15.     store(creator, library, evaluation_traces)

The key invariant enforced by this design is only-the-library-changes: within a (stack, benchmark) cell, task set, model, and harness are held fixed across all four conditions, so any pass/fail delta is attributable to which skill library (if any) is in context.

3.2 Why three creators from the same signals produce different libraries

This is a design choice worth pulling apart, because it’s the one place the paper explicitly measures how authoring process, not just skill content, shapes outcomes — why it matters, the obvious alternative, and where each approach breaks:

The anthropic creator follows Claude Code’s skill-authoring guide and is measurement-driven: draft a skill → run the agent on test prompts with and without it → grade both → rewrite from the measured difference → repeat until the skill demonstrably helps on its own benchmarked loop. Why it works: it directly optimizes for the thing you eventually care about (does this skill help?), so a skill that never clears its own internal bar gets rejected before it ever reaches the eval run. The obvious alternative is to trust the skill author’s intuition without a measurement loop — cheaper, but blind to exactly the kind of regression this paper studies. Where it fails: the internal benchmarked loop tests each skill on a handful of curated prompts, which is a much smaller and possibly unrepresentative sample than the full downstream evaluation; a skill can pass its own internal check and still regress differently on out-of-loop tasks it wasn’t tested against.
The openai creator follows the Codex skill guide and is single-pass: scaffold the skill files, validate their structure, emit them — no eval loop, no measurement of effect. Why it’s attractive: it’s fast and requires no benchmarking infrastructure, which matches how most practitioners actually author skills in the wild (write it once, ship it). The obvious alternative is the anthropic-style measurement loop; the tradeoff is authoring cost versus blind-spot risk. Where it fails: with no measured feedback, a skill’s text is only ever checked for internal structural validity, never for behavioral effect, which is exactly the setup this paper shows leads to disproportionately more regressions on some stacks (Table 2: openai’s SpreadsheetBench nets are consistently the weakest or tied-weakest of the three).
The “Ours” creator (the paper’s own contribution) adds discover-before-create (search the library for a similar existing skill and update it rather than duplicating) and a self-critique pass that rewrites the draft against a checklist, using only standard-library helpers so it is harness-agnostic. Why it works: discover-before-create directly targets one failure mode — proliferating near-duplicate, possibly conflicting skills — that the other two creators don’t explicitly guard against; the self-critique pass is a cheap internal consistency check without needing a full benchmarked eval loop. The obvious alternative is either of the above two extremes; Ours sits in between (cheaper than a full measurement loop, more scrutinized than a single pass). Where it fails: self-critique against a checklist is still just the model checking its own work — it has no access to ground-truth outcomes, so it cannot catch a skill that reads as internally coherent but is behaviorally wrong on real tasks (this is visible in Section 5.3’s UID0025 case, where the Ours library’s re-grounded formula still mis-anchors a cell range and fails to recover the answer that anthropic’s library does recover).

The paper is explicit that it does not rank these three creators and no conclusion depends on which one authored a given skill — they exist purely to generate multiple libraries from one evidence set so the study isn’t reporting an artifact of one author’s writing style.

3.3 Outcome taxonomy: four buckets per task

Every task, compared between its none run and its treatment run, falls into exactly one of four buckets:

Outcome	none → treatment	What it means
Gain	fail → pass	The library fixed something the bare agent got wrong.
Regression	pass → fail	The library broke something the bare agent got right.
Residual failure	fail → fail	Neither the bare agent nor the library solves it.
Retained	pass → pass	The library didn’t touch a task the bare agent already solved.

flowchart TB
    subgraph WithoutSkills["Without skills"]
        P0["Pass"]
        F0["Fail"]
    end
    subgraph WithSkills["With skill library"]
        P1a["Pass"]
        F1a["Fail"]
        P1b["Pass"]
        F1b["Fail"]
    end
    P0 -->|"stays correct"| P1a["Retained"]
    P0 -->|"broke it"| F1a["Regression"]
    F0 -->|"fixed it"| P1b["Gain"]
    F0 -->|"still wrong"| F1b["Residual failure"]

Figure 4 (math/taxonomy-visualizing figure): the four-way pass/fail transition table underlying every number in the paper. The aggregate pass-rate delta reported by ordinary evaluation collapses this whole 2×2 structure down to a single scalar, $c - b$ (gains minus regressions), which is exactly why two libraries with the same aggregate can have wildly different Gain/Regression breakdowns (Section 6).

3.4 The mechanism-classification procedure for regressions

For every regression (pass under none, fail under treatment), the authors read the paired trajectories and assign one label using this decision procedure:

1.  read the `none` trajectory and the treatment trajectory for this task
2.  was any skill body invoked or read in the treatment run?
3.  IF NOT invoked:
4.      check for influence evidence:
5.          (a) does the SAME task flip the SAME way under >=2 independently-authored libraries?
6.          (b) does the behavioral change track vocabulary named in a skill's DESCRIPTION
                (not body, since body was never read)?
7.      IF (a) or (b) holds: label = OSMOSIS
8.      ELSE: label = OTHER (presence-only flip, no influence evidence — treated as noise)
9.  IF invoked or read:
10.     locate the FIRST stage (reading the input vs. checking the output) where the
             treatment trajectory diverges from the (correct) `none` trajectory
11.     IF divergence is at the INPUT stage (wrong table / range / entity / definition / vintage):
12.         label = GROUNDING DISPLACEMENT
13.     IF divergence is at the OUTPUT stage (a check the baseline ran is now
             suppressed or replaced):
14.         label = VERIFICATION DISPLACEMENT
15.     IF divergence spans BOTH stages: label = GROUNDING + VERIFICATION

Two design choices in this procedure deserve explicit discussion:

Requiring cross-library reproduction or description-vocabulary matching before labeling osmosis (steps 5–8) is a deliberate high bar. Why: a single library flipping a single task on a single run could just be normal LLM sampling variance, since the paper does not replicate across random seeds (Section 6.2 flags this explicitly as a limitation). Requiring the same task to flip the same way under multiple independently-written libraries — or requiring the model’s own reasoning trace to echo the vocabulary of a skill’s description — is a cheap but real falsification test: it is a low-probability coincidence for three separately-authored descriptions to happen to nudge the exact same wrong answer unless the description is genuinely causal. The alternative — labeling every presence-only flip as osmosis — would inflate the mechanism’s apparent frequency with pure noise; the paper explicitly reserves an “Other” bucket for these unproven flips (4 of 81 OfficeQA-Pro regressions).
A single author assigned all labels (stated directly in Section 5). This is the most defensible-but-real weakness of the whole classification: there is no inter-annotator agreement statistic, and a solo coder’s threshold for “does this evidence look like influence” is exactly the kind of judgment call that benefits from a second, blinded coder. The paper is honest about this in its own limitations section, and I flag it again in Section 8 below because it’s the single biggest thing I’d want fixed before trusting the mechanism proportions (as opposed to the qualitative existence of each mechanism, which the worked cases in Appendices B–C make hard to dispute).

4. The core statistical decomposition, derived

4.1 Net effect is exactly gains minus regressions

Let $N$ be the number of tasks in a benchmark, $c$ the number of Gain transitions (fail→pass), $b$ the number of Regression transitions (pass→fail), $r$ the number of Residual failures, and $t$ the number Retained. By construction:

N = c + b + r + t \tag{2}

The pass rate without skills is $p_0 = \dfrac{t + b}{N}$ (everything that passed under none: Retained plus what will later regress). The pass rate with skills is $p_1 = \dfrac{t + c}{N}$ (everything passing under treatment: Retained plus new Gains). The paired difference is:

\Delta = p_1 - p_0 = \frac{(t + c) - (t + b)}{N} = \frac{c - b}{N} \tag{3}

This is the derivation behind the paper’s central methodological point: the aggregate pass-rate delta is already a normalized net effect — it is mathematically identical to $(c-b)/N$ , so it necessarily discards the individual magnitudes of $c$ and $b$ whenever they’re reported without the paired counts. Two libraries can have the exact same $\Delta$ with wildly different $(c, b)$ pairs — e.g. $(c,b) = (10, 2)$ and $(c,b) = (50, 42)$ both give a net of $+8$ , but the second library is churning through five times more of the task population to get there, which matters enormously if you care about reliability rather than average improvement. This is precisely the anthropic-vs-openai contrast on Claude Code · sonnet-4.6 / OfficeQA-Pro in Section 5: gains of 10 vs 12, but regressions of 2 vs 7, so ranking by gains alone (openai first) reverses under ranking by net effect (anthropic first).

4.2 McNemar’s exact test, worked through

For each treatment-vs-none pair, we only need the discordant counts $b$ (pass→fail) and $c$ (fail→pass); concordant tasks (both pass or both fail) carry no information about directional effect and are dropped from the test. Under the null hypothesis that the library has no systematic directional effect, each discordant task is a fair coin flip between “was a gain” and “was a regression,” i.e. $b \sim \text{Binomial}(b+c, 0.5)$ under $H_0$ . The exact two-sided p-value is:

p = 2 \times \min\left(\sum_{k=0}^{b} \binom{b+c}{k} 0.5^{b+c}, \; \sum_{k=b}^{b+c} \binom{b+c}{k} 0.5^{b+c}\right) \tag{4}

capped at 1. Intuitively: if $b$ and $c$ are close to equal (many discordant tasks each way), the observed split looks like a fair coin, so $p$ stays large — the library has some effect on many tasks, but no reliable directional bias toward improvement. Only when the split is lopsided (e.g. $b=16, c=59$ on Claude Code · sonnet-4.6 / SpreadsheetBench for the anthropic library) does the exact binomial tail shrink toward significance. This is why a library can move a large fraction of tasks (both up and down) and still fail to clear significance — the direction is what’s tested, not the volume of change.

4.3 The 95% Newcombe confidence interval on the paired difference

Alongside the p-value, the paper reports a Newcombe-style confidence interval on $\Delta = (c-b)/N$ , which accounts for the paired (non-independent) structure of the two proportions $p_0$ and $p_1$ rather than treating them as two unpaired binomial samples (which would understate the precision available from a matched design). The practical read: when this interval straddles zero (13 of 18 conditions in Table 2), the observed net effect is statistically indistinguishable from what paired sampling noise alone could produce — a caution the paper is careful to keep visible in every row of its main results table rather than reporting only point estimates.

4.4 Bonferroni correction across the 18 comparisons

With $m=18$ simultaneous McNemar tests, the corrected significance threshold is $\alpha/m = 0.05/18 \approx 0.0028$ . Of the 5 conditions that clear the uncorrected $p<0.05$ bar, only 3 clear the corrected bar — all three are the Claude Code · sonnet-4.6 / SpreadsheetBench trio (all three libraries, $p<.001$ even after correction). The other two nominally-significant cells (Claude Code · sonnet-4.6 / OfficeQA-Pro’s anthropic library at $p=.039$ , and OpenCode · minimax-m2.7 / SpreadsheetBench’s anthropic library at $p=.008$ ) do not survive correction (adjusted $p = 0.69$ and $0.14$ respectively). This is the paper turning its own statistical machinery against its most favorable-looking numbers, and it is the single most important number in the whole results section to internalize: out of 18 attempted demonstrations that “adding a skill library reliably helps,” only 3 — confined to one stack and one benchmark — hold up.

5. Experiments and results

5.1 The headline table: gains, regressions, net effect across 18 conditions

The full results (recreated from Table 2 of the paper) span 3 stacks × 2 benchmarks × 3 libraries:

Stack / Benchmark	Library	Pass rate (no skills → with skills)	Gains	Regressions	Net	95% CI	p
OpenCode·minimax / OfficeQA-Pro	anthropic	51.1→52.1	14	13	+1	(−9.8,+11.9)	1.000
OpenCode·minimax / OfficeQA-Pro	openai	51.1→55.3	16	12	+4	(−6.7,+15.3)	0.572
OpenCode·minimax / OfficeQA-Pro	Ours	51.1→60.6	15	6	+9	(+0.2,+18.9)	0.078
Codex·gpt-5.4-mini / OfficeQA-Pro	anthropic	54.3→56.4	17	15	+2	(−9.7,+13.9)	0.860
Claude Code·sonnet-4.6 / OfficeQA-Pro	anthropic	72.3→80.9	10	2	+8	(+1.5,+15.5)	0.039
Claude Code·sonnet-4.6 / OfficeQA-Pro	openai	72.3→77.7	12	7	+5	(−3.7,+14.3)	0.359
OpenCode·minimax / SpreadsheetBench	anthropic	63.3→70.2	62	35	+27	(+2.0,+11.8)	0.008
OpenCode·minimax / SpreadsheetBench	openai	63.3→64.3	45	41	+4	(−3.6,+5.7)	0.747
Claude Code·sonnet-4.6 / SpreadsheetBench	anthropic	70.2→81.1	59	16	+43	(+6.8,+15.2)	<.001
Claude Code·sonnet-4.6 / SpreadsheetBench	openai	70.2→81.1	63	20	+43	(+6.5,+15.4)	<.001
Claude Code·sonnet-4.6 / SpreadsheetBench	Ours	70.2→82.1	66	19	+47	(+7.5,+16.4)	<.001

(Bolded rows survive Bonferroni correction. Rows omitted for space follow the same pattern; see paper Table 2 for the full 18.)

Summed across all 18 conditions: 553 gain transitions, 324 regression transitions. Regressions offset 59% of gross gains, leaving a retained net of 229. Split by benchmark: OfficeQA-Pro gained 122 and broke 81 (66% cancelled); SpreadsheetBench gained 431 and broke 243 (56% cancelled). This is the “regression tax” the paper is named for — and note it is not concentrated in weak cells: even a library with a large positive net, like OpenCode·minimax’s openai library on SpreadsheetBench (broke 41, gained 45), is churning through nearly as many regressions as gains.

5.2 Mechanism breakdown: OfficeQA-Pro (81 regressions, fully coded)

Mechanism	minimax-m2.7	gpt-5.4-mini	sonnet-4.6	All	%
Grounding displacement	23	34	2	59	72.8%
Osmosis	5	0	9	14	17.3%
Grounding + verification	1	2	0	3	3.7%
Other	2	1	2	5	6.2%

pie title OfficeQA-Pro regression mechanisms (N=81)
    "Grounding displacement" : 59
    "Osmosis" : 14
    "Grounding + verification" : 3
    "Other" : 5

Figure 5 (reproduced experimental result, paper Table 3): grounding displacement dominates when a skill body is actually engaged (23/31 on minimax-m2.7, 34/37 on gpt-5.4-mini); osmosis concentrates on the stack where the skill tool is invoked least often relative to regressions (sonnet-4.6, 9 of 13). This split by stack is itself informative — it shows the mechanism a regression falls under is largely a function of how often that stack’s harness chooses to invoke skills at all, not of skill content.

5.3 Mechanism breakdown: SpreadsheetBench (243 regressions, coarser coding)

Mechanism	minimax-m2.7	gpt-5.4-mini	sonnet-4.6	All	%
Osmosis	45	25	0	70	28.8%
Body engaged	6	0	40	46	18.9%
Grader artifact	10	17	5	32	13.2%
Other	54	31	10	95	39.1%

pie title SpreadsheetBench regression mechanisms (N=243)
    "Osmosis" : 70
    "Body engaged" : 46
    "Grader artifact" : 32
    "Other" : 95

Figure 6 (reproduced experimental result, paper Table 4): SpreadsheetBench regressions cannot be cleanly split into grounding vs. verification the way OfficeQA-Pro’s can (cell-value answers don’t expose the first faulty stage as legibly as a single toleranced number does), so the paper reports a coarser breakdown. The grader artifact category (13.2%) is a genuinely important confound this table surfaces: some fraction of “regressions” are not behavioral at all — they’re the value-only grader failing to recompute a modern Excel function like AGGREGATE, scoring a correct formula as wrong.

5.4 The grader-artifact fix recovers up to 49 points of pass rate

Section 5.3 of the paper re-grades all treatment failures that hold a formula in the graded region (663 of them across all 9 treatment conditions) using a full spreadsheet engine that actually recalculates the formula, rather than the shallow value-only comparison. The recovery:

Stack / Library	Raw pass rate	Corrected pass rate	Task-conditions recovered
minimax-m2.7 / anthropic	70.2	74.7	+18
minimax-m2.7 / openai	64.3	68.4	+16
minimax-m2.7 / Ours	65.6	69.6	+16
gpt-5.4-mini / anthropic	67.6	78.6	+43
gpt-5.4-mini / openai	66.6	79.1	+49
gpt-5.4-mini / Ours	67.1	77.8	+42
sonnet-4.6 / anthropic	81.1	84.9	+15
sonnet-4.6 / openai	81.1	85.2	+16
sonnet-4.6 / Ours	82.1	84.9	+11

This single fix lifts the gpt-5.4-mini stack from the mid-60s into the high-70s in pass rate — a swing entirely attributable to a grading toolchain limitation, not to any change in agent behavior. It’s a striking demonstration that “regression” is not automatically synonymous with “the agent did something wrong”: 226 of the 663 formula-bearing failures (34%) were already-correct outputs the shallow grader could not verify. This directly motivates one of the paper’s two headline recommendations: skill libraries should carry executable output checks, not just procedural steps, precisely because so much of the recoverable failure mass sits at the verification stage rather than the method stage.

6. Worked cases: why this taxonomy is falsifiable, not vibes

I want to walk through two of the paper’s worked cases in detail because they’re what convinced me the mechanism labels are more than post-hoc storytelling.

UID0025 (grounding displacement). Task: absolute difference in U.S. public-works spending between 1934 and 1946, ground truth 142. Without skills, the agent reads 549 (1934) and 407 (1946), computes $|549-407|=142$ , and passes. With the anthropic library, two navigation-and-arithmetic skills get invoked; the agent follows their prescribed navigation path to a different pair of figures — 949 instead of 549 for 1934 — and computes $|949-407|=542$ , an entirely wrong answer produced by entirely correct arithmetic on the wrong input. The method is intact. The regression is 100% at the input stage. This is the pattern behind 59 of 81 OfficeQA-Pro regressions: a general-purpose navigation procedure can override a correct reading the bare agent would have made on its own, because the procedure doesn’t know it’s being applied to a case where the bare agent already had it right.

UID0096 (osmosis). Task: centered moving average of a customs-duty rate, ground truth 0.377 (37.7%). Without skills, sonnet-4.6 answers 37.708% and passes. With any of the three libraries present — and, critically, with the skill tool invoked zero times across all three treatment runs — the agent answers 38.757%, identical to three decimal places, under every library. No skill body was ever read. The only thing that changed between the passing run and the three failing runs is which skill descriptions sat in the system prompt. The paper’s evidence that this is real influence and not sampling noise: (1) the exact same wrong value (38.757%) reproduces across three independently-authored libraries — a coincidence this precise is implausible under pure decoding variance; (2) the shift tracks specific vocabulary (“revised,” “customs”) that recurs across the three libraries’ skill descriptions, even though none of the bodies carrying detailed procedures were ever loaded. This is the mechanism no retrieval-gated or invocation-gated defense method can see, because those methods only intervene when a skill is actually invoked — here it never was.

Both cases hold the visible reasoning (arithmetic, spreadsheet logic) constant and isolate the divergence to one stage, which is exactly the discipline that separates this from a purely qualitative “sometimes skills seem to hurt” observation.

7. Design-choice discussion: what else could the authors have done, and why didn’t they?

Why paired trajectory-reading instead of automated diffing? An automated pipeline diffing traces token-by-token could scale to more regressions and remove the solo-coder bias (Section 3.4). The obvious cost: automated diffing cannot judge semantic stage (is this divergence at the “reading the input” stage or “checking the output” stage?) without something close to another LLM judge, which reintroduces exactly the kind of unverified black-box judgment the paper is trying to avoid by hand-coding a small, well-understood sample (81 + 243 regressions, still substantial).
Why not run multiple seeds to estimate variance? The paper explicitly flags (Section 6.2) that it does not replicate across random seeds, so run-to-run stochastic variance is not separated from library-induced variance. The alternative — running each condition $k$ times and averaging — would multiply the run budget by $k$ (already at 5,832 runs) and raise real cost, but it is the single most important addition I’d want in a follow-up: without it, some fraction of the “Other” bucket (5 + 95 = 100 unresolved regressions across both benchmarks) could plausibly be decoding noise rather than a fourth undiscovered mechanism.
Why couple harness and model per stack rather than crossing all combinations? Crossing 3 harnesses × 3 models would be 9 stacks instead of 3, tripling the run budget again, and — more importantly — most of those combinations (e.g. running Claude Code against MiniMax) aren’t how these harnesses are used in practice, since each harness typically ships tuned for its native model. The tradeoff the paper accepts: model and harness effects are confounded, so “sonnet-4.6 shows the strongest osmosis on OfficeQA-Pro” could be a Claude Code artifact, a Claude-model artifact, or both — the paper cannot separate these, and says so.
Why McNemar’s test rather than a mixed-effects logistic regression across all 18 conditions jointly? A single hierarchical model could borrow statistical strength across stacks and benchmarks and potentially detect weaker but real effects that per-cell McNemar tests (each starved of power at $N \approx 94$ or $392$ ) miss. The tradeoff: a joint model requires assumptions about how effects generalize across stacks that the paper’s own confound (Section above) makes hard to justify, and per-cell exact tests are more transparent and harder to misuse than a black-box mixed model’s coefficients. Given how much of this paper’s credibility rests on not over-claiming, the more conservative, per-cell choice fits the paper’s own epistemic stance.

8. Limitations

The paper is unusually forthcoming about its own limitations (Section 6.2), and they’re worth restating precisely rather than glossing over:

Domain scope: both benchmarks are office-automation tasks where grounding (reading the right input) and output format are prominent failure surfaces. The paper does not claim these mechanism proportions transfer to domains where “the method itself is the main bottleneck” — e.g. multi-step mathematical proof search or complex code refactoring, where an agent’s procedural reasoning, not its ability to locate the right cell in a spreadsheet, may be the dominant failure mode.
Model–harness confound: as discussed above, each stack pairs one harness with its native model, so effects attributed to “the stack” cannot be cleanly split into “caused by the model” vs. “caused by the harness.”
Statistical fragility: only 3 of 18 comparisons survive Bonferroni correction, and all three are confined to a single stack (Claude Code · sonnet-4.6) and a single benchmark (SpreadsheetBench). The paper’s own framing is that this narrowness reinforces the regression-tax thesis (most apparent improvements are statistically indistinguishable from noise once corrected for multiple comparisons) rather than undermining the paper’s central claim — but it does mean the specific numeric claims (e.g. “+43 net effect”) should not be read as broadly generalizable facts about skill libraries in general.
Solo-coder labeling: as discussed in Section 3.4, mechanism labels for all 324 regressions were assigned by a single author without a documented inter-coder agreement check.
No replication across seeds: run-to-run stochastic variance in LLM sampling is not measured or subtracted out, so some fraction of observed flips (particularly in the unresolved “Other” bucket) could be decoding noise rather than a genuine mechanism.

9. Critical analysis

(a) Weaknesses and flaws specific to this paper. The single most consequential methodological gap is the missing seed-replication (Section 8, item 5). The paper’s entire regression-mechanism story hinges on being able to attribute a pass→fail flip to the skill library rather than to ordinary LLM sampling variance, and the only defense offered against this alternative explanation is the cross-library reproduction check used for osmosis labeling (Section 3.4) — but that check is applied only to the osmosis category, not to grounding or verification displacement regressions, which make up the large majority (59+3 of 81 on OfficeQA-Pro, and an unknown fraction of the 243 SpreadsheetBench regressions). It is entirely possible that some non-trivial fraction of grounding-displacement-labeled regressions are actually a case where the bare agent’s correct answer on none was itself a lucky sample, and a re-run without skills would have failed too. Without at least a handful of repeated none runs to estimate this baseline noise floor, the mechanism proportions (72.8% grounding, 17.3% osmosis on OfficeQA-Pro) should be read as upper-bound estimates conditioned on the labeling scheme, not as ground truth about the underlying causal process.

(b) Limitations the authors understate or omit. The paper is admirably candid about domain scope, confounds, and statistical fragility, but it under-discusses one thing: skill library size is a confound the paper measures but doesn’t control for. Section 3.3 notes in passing that “libraries range from 3 to 23 skills” and are “not length-matched,” and explicitly says treatment-vs-treatment contrasts are “not a controlled token dose.” But this means every treatment-vs-none comparison in the main results table (Table 2, my Section 5.1) is also not a controlled token dose — a library with 23 skills adds vastly more context than one with 7, and “more context, unrelated to the task, hurts reasoning” is a well-established effect the paper itself cites (Shi et al. on GSM-IC distractors, Liu et al. on “Lost in the Middle”). It is entirely plausible that some fraction of what gets labeled “grounding displacement” or “osmosis” is really just a generic distractor-context effect that would occur with any 23 skills’ worth of irrelevant text, not specifically because of what those particular skills say. The paper never runs a length-matched control (e.g. 23 near-random or scrambled “placebo” skills) that would let it separate “content-specific displacement” from “generic long-context dilution.” This is a natural, cheap follow-up experiment the paper’s own related-work citations (Section 2) all but demand, and its absence weakens the strength with which the grounding-displacement mechanism, specifically, can be attributed to skill content rather than skill bulk.

(c) Concrete, specific improvement suggestions. First, run a length-and-token-matched placebo condition (random unrelated text of matched token count in place of a real skill library) in at least the two stacks/benchmarks that show significant effects, to separate content-driven displacement from generic context-dilution. Second, replicate the none condition with at least 3 seeds on a stratified sample of tasks (even 50–100 tasks would suffice) to estimate the baseline flip rate from sampling noise alone, which would let every subsequent mechanism-attribution claim carry an explicit noise-floor correction. Third, extend the mechanism-classification scheme with a second, blinded coder on at least a 20% sample of the 324 regressions and report Cohen’s kappa — this is cheap (a few hours of a second author’s time) relative to how much weight the paper’s headline numbers put on the labeling scheme being reliable. Fourth, given how large the grader-artifact confound turned out to be on SpreadsheetBench (32 of 243 regressions, and 226 of 663 formula failures more broadly), the paper should re-audit whether a similar “grading toolchain, not agent behavior” artifact could be inflating any OfficeQA-Pro regression counts — the paper checks this carefully for SpreadsheetBench but doesn’t apply the same scrutiny in reverse to the benchmark it treats as “clean.”

10. Reproducibility notes

Model–harness stacks: OpenCode+MiniMax-M2.7, Codex+GPT-5.4-mini, Claude Code+Claude Sonnet 4.6 — all closed or semi-closed models/harnesses as of the paper’s writing, meaning exact reproduction requires access to the same model checkpoints (subject to drift/versioning over time) and the same harness versions.
Benchmarks: OfficeQA-Pro is described as a curated, validated subset the authors built from U.S. Treasury documents (not a pre-existing public benchmark with a fixed release), so exact task-level reproduction depends on the authors releasing their curated subset; SpreadsheetBench is a public NeurIPS 2024 benchmark (Ma et al.) with a known grading-engine limitation the paper documents in detail (Section 3.1 of the paper, my Section 1.2) — anyone re-running this benchmark should be aware of the AGGREGATE/structured-reference blind spot before trusting raw pass-rate numbers.
Statistical protocol: exact two-sided McNemar test on discordant pairs, Newcombe 95% CI on paired difference, Bonferroni correction at $\alpha/18$ . All of this is fully specified and standard — a careful reader can re-derive every reported p-value and CI from the raw $(b,c,N)$ triples in Table 2 alone.
What’s not released/specified: the mechanism-classification labels (Section 5 of the paper) were assigned by one author reading paired trajectories; the paper does not state whether the full 324 raw trajectory transcripts are released for independent re-labeling, which would be the single most valuable artifact for anyone trying to validate the mechanism-proportion claims independently.

10.5 A worked comparison: same signals, three different regression profiles

To make Section 3.2’s abstract discussion of the three creators concrete, it’s worth walking through what happens to the same failure signal set on the OpenCode·minimax-m2.7 stack. The analyst stage (Section 3.1) extracts one shared pool of recurring failure causes from the baseline trajectories. The anthropic creator’s measurement-driven loop condenses these into a comparatively conservative set of skills, each individually vetted against its own benchmarked test prompts before being kept. The openai creator’s single-pass pipeline, with no eval loop, expands the same signal pool into 23 skills on this stack — nearly every recurring signal becomes its own skill, with no consolidation or rejection step. The Ours creator’s discover-before-create step merges overlapping signals, landing at 7 skills for the same evidence.

The downstream effect on SpreadsheetBench for this stack (Table 2, my Section 5.1) is instructive: anthropic nets +27 (survives at $p=.008$ uncorrected, though not after Bonferroni), openai nets only +4, and Ours nets +9. The 23-skill openai library is not obviously better for having more skills — if anything, its higher regression count relative to its gain count (41 regressions against 45 gains, versus anthropic’s 35 against 62) is consistent with the context-dilution concern raised in Section 9(b): more skill descriptions sitting in context, most never invoked, is more surface area for osmosis-style interference, and Table 4 (Section 5.3) shows osmosis is in fact the single largest regression mechanism on this exact stack (45 of 115 minimax-m2.7 SpreadsheetBench regressions). This doesn’t prove length alone is the culprit — content still matters, since Ours also merges related signals into fewer skills and ends up in between — but it is a clean illustration of why the paper insists library size and authoring process, not just skill content, need to be reported alongside any net-effect number.

11. Takeaways

If you ship agent skills, tool-use guides, or any in-context procedural scaffold in production: (1) report paired gain/regression counts, not just aggregate pass-rate deltas — two libraries with identical net effect can be trading away wildly different amounts of prior reliability; (2) explicitly test for presence-only (osmosis) effects by running a condition where descriptions are present but bodies are inaccessible, since retrieval-gated or invocation-gated defenses cannot see this channel at all; (3) invest disproportionately in grounding and verification content over procedural “how-to” content — this paper’s data says existing skills already over-serve the procedural middle of the pipeline and under-serve the two ends where regressions and residual failures concentrate. None of this requires new infrastructure beyond what the paper already used: a paired run harness and a willingness to read the failures your gains are quietly buying.