SEED: When Hindsight Becomes a Training Signal, Not Just a Better Prompt

Review date: 2026-07-20 Review author: Zhongzhu Zhou Paper reviewed: SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning Paper authors: Jinyang Wu, Shuo Yang, Zhengxi Lu, Fan Zhang, Yuhao Shen, Lang Feng, Haoran Luo, Zheng Lian, Shuai Zhang, Zhengqi Wen, Jianhua Tao arXiv: 2607.14777 Status: Preprint, submitted 16 July 2026

Short Answer

Long-horizon agentic reinforcement learning has an awkward asymmetry: the reward signal you actually get is a single number per episode (“did the agent finish the task or not”), but the thing you are trying to train is a sequence of hundreds of individual token-level decisions spread across dozens of environment turns. Standard outcome-based RL algorithms like GRPO paper over this mismatch by broadcasting the same scalar advantage to every token in a trajectory — every action in a winning trajectory gets reinforced equally, every action in a losing trajectory gets discouraged equally, regardless of whether that specific action was actually the reason for the outcome. SEED (SElf-Evolving on-policy Distillation) attacks this mismatch directly, using a mechanism that is conceptually simple but carefully engineered in its details: after a trajectory completes, feed the entire completed trajectory back into the same model, ask it to summarize what it learned as a short natural-language “hindsight skill” (a reusable workflow, a decisive observation, or a failure-avoidance rule), then re-score the exact same already-sampled tokens under two different contexts — once with the ordinary history, once with the hindsight skill spliced in — and use the resulting shift in the model’s own token probabilities as a dense, per-token training target. Because the skill only ever gets inserted during training (never at inference time), the model has to learn to act as if it already knew the hindsight lesson, without ever being told the lesson explicitly at deployment. The self-evolving part is what stops this being a static distillation setup: at every policy-update round, the exact same, just-updated model checkpoint is used both to collect new trajectories and to analyze them into skills, so as the actor’s failure modes and successes shift over training, the analyzer’s hindsight commentary shifts with it — there is no separately-trained, frozen skill generator. Averaged across ALFWorld, WebShop, and seven search-based QA benchmarks and three backbone sizes (Qwen2.5-3B/7B-Instruct, Qwen3-1.7B-Instruct), SEED improves ALFWorld’s macro-average success rate from 75.0 (GRPO) to 91.8, WebShop’s success rate from 63.3 to 78.9, and Search-QA accuracy from 36.4 to 45.7 — and, crucially, it beats every other tested self-distillation baseline (OPSD, Skill-SD, RLSD, SDAR) that also uses a privileged self-teacher, isolating the specific contribution of making the teacher’s supervision on-policy and self-evolving rather than static.

Key Takeaways

• The core supervision-gap problem SEED targets is real and easy to underestimate: GRPO’s group-relative advantage is a single scalar per trajectory, broadcast identically to every valid token — when two trajectories in the same rollout group get the same terminal reward (a common occurrence, especially as training converges), that scalar is literally zero for both, and the RL gradient vanishes for every token in both trajectories, even though one trajectory may have taken a much cleaner, more efficient path to the same outcome.
• SEED’s two-stage design cleanly separates “teach the model how to introspect” (Stage 1: hindsight-skill SFT on offline trajectories, annotated by an external analyzer, GLM-5.2) from “use that introspection ability as a live training signal” (Stage 2: self-evolving on-policy distillation, where the model analyzes its own current trajectories, no external analyzer needed).
• The mathematical core is a confidence-gated, sigmoid-weighted on-policy distillation loss (Eq. 1) whose expected gradient, proven in Proposition 1, is provably equivalent to KL-distillation toward a skill-reweighted target distribution built entirely from the current policy’s own token-context occupancy — this is the theoretical content behind the paper’s claim that the auxiliary signal is “occupancy-matched.”
• Proposition 2 makes a sharper, testable claim: even when every trajectory in a rollout group is tied in outcome (so GRPO’s advantage and gradient are provably zero), the OPD gradient’s squared norm equals the variance of the hindsight gate across candidate tokens — meaning it is provably nonzero exactly when the hindsight skill actually discriminates between good and bad token choices, giving SEED credit-assignment power in precisely the regime where outcome-only RL goes silent.
• Ablations (Table 2) show all three components matter and none is redundant: removing hindsight-skill SFT costs 5.8 points, removing the self-evolving refresh costs 4.8 points, and replacing on-policy skills with a static offline skill library costs 7.4 points — the largest single drop, underscoring that whose trajectories the skill comes from matters more than having a skill at all.
• Perhaps the most practically important empirical finding is the Skill-Prompt vs. SEED comparison: supplying the same kind of hindsight skill as an inference-time prompt (no training) underperforms SEED on every single aggregate metric, across all three backbones — internalizing the skill into the weights via distillation beats handing the model the same information as text at test time, which has real implications for how much “prompt engineering effort” should really be seen as a substitute for training-time distillation.

Prerequisites: What You Need to Know First

This is an agentic reinforcement learning paper, sitting at the intersection of RL for LLMs, on-policy knowledge distillation, and hindsight/experience-replay ideas originally from classical RL. To follow the method section without hand-waving, you need five pieces of background: what a partially observable Markov decision process (POMDP) is and how it maps onto a multi-turn LLM agent, what GRPO’s group-relative advantage actually computes and why it is a single scalar per trajectory, what “on-policy” means and why distribution mismatch between training data and the current policy is a real cost (not a pedantic distinction), what knowledge distillation is and what makes a distillation signal “on-policy” versus “off-policy,” and what a confidence gate / sigmoid weighting mechanism is doing when it is applied to a log-probability difference. I build these up one at a time below.

Framing a Multi-Turn LLM Agent as a POMDP

A long-horizon agentic task — cooking in a simulated kitchen (ALFWorld), shopping on a simulated e-commerce site (WebShop), or answering a question by issuing search queries (Search-QA) — is naturally modeled as a partially observable Markov decision process $(\mathcal{S}, \mathcal{A}, \mathcal{O}, \mathcal{T}, \Omega, \mathcal{R}, \gamma)$: a latent state space $\mathcal{S}$ (the true, possibly hidden, world state — e.g., where every object in the kitchen actually is), an action space $\mathcal{A}$ (textual responses or executable tool calls), an observation space $\mathcal{O}$ (what the agent actually sees — e.g., the text description returned after moving to a location), a transition kernel $\mathcal{T}$ (how the hidden state evolves given an action), an observation kernel $\Omega$ (how observations are generated from the hidden state), a reward function $\mathcal{R}$, and a discount factor $\gamma$. At each timestep $t$, the agent maintains an interaction history $h_t = (o_0, a_0, o_1, a_1, \dots, o_t)$ — the entire textual transcript so far — and samples its next action from $a_t \sim \pi_\theta(\cdot \mid h_t)$: the LLM, conditioned on the whole transcript, produces the next action as if it were just another piece of text to continue. This is why LLM-agent training reduces to a language-modeling problem with an unusual reward structure: the “sequence” being generated interleaves the model’s own outputs with externally-injected observation text, and the reward $R(\tau)$ for a completed trajectory $\tau = \{(o_t, a_t, r_t)\}_{t=0}^{T-1}$ is typically sparse — a binary success/failure signal that only becomes available once the whole episode ends, not per-token or even per-turn.

GRPO’s Group-Relative Advantage: One Number Per Trajectory

GRPO (Group Relative Policy Optimization) is the outcome-based RL backbone SEED builds on top of. For a task prompt $q$, GRPO samples a group of $N$ trajectories from the current (frozen) policy, $\{\tau_q^{(1)}, \dots, \tau_q^{(N)}\}$, and computes the group’s empirical mean $\mu_q$ and standard deviation $\sigma_q$ of the trajectory-level outcomes $R(\tau_q^{(n)})$. The advantage for trajectory $n$ is then

$$A^{\text{rl}}_{q,n} = \frac{R(\tau_q^{(n)}) - \mu_q}{\sigma_q + \epsilon}. \tag{RL-Adv}$$

The crucial detail, easy to gloss over: this advantage is a single scalar per trajectory, and it gets broadcast identically to every valid action token in that trajectory — $A^{\text{rl}}_{q,n,t,\ell} = A^{\text{rl}}_{q,n} \cdot m_{q,n,t,\ell}$, where $m$ is just a validity mask. A trajectory that succeeded after 30 confused, meandering steps gets exactly the same per-token reward signal as a trajectory that succeeded in 5 clean, efficient steps — GRPO has no built-in way to say “this specific action, at this specific step, was the good decision; that other one, three turns earlier, was a wasted detour.” This is the precise mathematical form of the “supervision gap” the paper’s introduction describes in prose: the reward’s granularity (one number per episode) does not match the granularity of the thing being optimized (one policy decision per token).

On-Policy vs. Off-Policy, and Why Distribution Mismatch Is a Real Cost

“On-policy” means the training data (trajectories, or in SEED’s case, the hindsight skills derived from trajectories) was generated by the current version of the policy being trained, as opposed to an older checkpoint or a completely separate model. Why does this matter beyond textbook correctness? Concretely: if you generate hindsight skills once, early in training, using an early (weak) policy checkpoint, those skills describe the failure modes and successful strategies of that early policy — as training progresses and the policy improves, it stops making those particular mistakes and starts making different ones, so the frozen early-stage skills become progressively less relevant, or actively misleading, supervision. SEED’s ablation table quantifies this precisely: replacing on-policy skill generation with a static offline skill library (generated once, from an earlier snapshot) costs 7.4 points on ALFWorld — the single largest ablation drop in the paper, larger than removing hindsight-skill SFT entirely. This is the empirical evidence behind the theoretical staleness bound (Proposition 3) discussed below.

On-Policy Distillation: Teacher and Student Sharing the Same Weights

Classical knowledge distillation trains a smaller “student” model to imitate a larger, separately-trained “teacher” model’s output distribution. On-policy distillation (the specific variant SEED builds on, following prior work like OPSD, SDAR, and RLSD) changes two things at once: first, the “teacher” and “student” are literally the same set of parameters $\pi_\theta$, evaluated under two different input contexts (one with privileged/hindsight information spliced in, one without); second, the tokens being scored are not freshly sampled from the teacher distribution — they are the tokens the model itself already sampled under the ordinary (student) context during rollout collection. This second property is what makes it “on-policy”: rather than teaching the student to imitate an idealized teacher output, the signal only ever touches actions the policy would actually take, re-scored under a slightly richer context. This design choice avoids introducing distribution shift from teacher-generated tokens the student would never naturally produce.

The Confidence Gate: Turning a Log-Probability Gap into a Soft 0-to-1 Weight

The central trick that converts “the skill changed the model’s opinion about this token” into a usable training signal is a sigmoid confidence gate. If $\ell^{\text{skill}}$ is the model’s log-probability for a sampled token under the skill-augmented context, and $\ell^{\theta}$ is its log-probability under the ordinary context, their difference $\Delta = \ell^{\text{skill}} - \ell^{\theta}$ is positive when the hindsight skill makes the sampled token more likely (the skill “endorses” this token) and negative when the skill makes it less likely (the skill “disapproves”). Passing this through a sigmoid, $g = \sigma(\beta_{\text{opd}} \Delta)$, converts an unbounded real number into a soft weight in $(0,1)$: strongly endorsed tokens get a gate close to 1, strongly disapproved tokens get a gate close to 0, and $\beta_{\text{opd}}$ (a hyperparameter, set to 5.0 in the paper) controls how sharply the gate discriminates between the two. This gate is exactly the “confidence” in “confidence-gated” — it measures how confidently the hindsight skill supports or opposes each individual token, and this confidence, not the raw reward, is what SEED distills back into the ordinary policy.

A Concrete Walk-Through of the ALFWorld Episode Structure

To ground all of the formalism above in something tangible, it helps to trace through what a single ALFWorld episode actually looks like as a sequence of $(o_t, a_t, r_t)$ triples, since this is the concrete instantiation of $\tau$ that gets fed into every equation in this review. Consider the task “put a candle in toilet” (the exact example used in the paper’s Figure 6, discussed at length below). At $t=0$, the observation $o_0$ is a natural-language room description (“You are in the middle of a room. Looking quickly around you, you see a bathtubbasin 1, a garbagecan 1, …”). The agent’s action $a_0$ is a textual command from a constrained but large admissible-action set (e.g., “go to shelf 1”). The environment then returns a new observation $o_1$ describing what happened and what is now visible (e.g., “You arrive at shelf 1. On the shelf 1, you see a soapbottle 2, and a spraybottle 1.”). This $(o_t, a_t)$ alternation continues — in the paper’s successful example, five steps total — until either the task is completed (giving $r_T = 1$, the sparse terminal reward) or the maximum interaction-step budget (30 for ALFWorld, per Table 5) is exhausted (giving $r_T=0$). Every intermediate reward $r_t$ for $t < T$ is $0$; there is no partial credit for individual actions, which is exactly the sparse-reward structure that motivates the entire method. When this full sequence completes, it becomes exactly the $\tau$ fed into the hindsight analyzer $\mathcal{A}_{\theta_{\text{old}}}(x_\tau)$, which reads through all five (or up to thirty) of these observation-action pairs plus the final outcome, and produces the natural-language hindsight skill discussed throughout this review.

Notation Reference Table

Because the method section introduces a fair number of subscripted symbols, here is a single table to refer back to while reading the derivations below.

Symbol	Meaning
$q$	a task prompt
$\tau_q^{(n)}$	the $n$-th sampled trajectory for task $q$
$h_{q,n,t}$	the ordinary interaction history at timestep $t$ of trajectory $n$
$\tilde h_{q,n,t}$	the skill-augmented history (hindsight skill spliced in)
$s_q^{(n)}$	the hindsight skill extracted from trajectory $\tau_q^{(n)}$
$\pi_{\theta_{\text{old}}}$	the frozen policy snapshot at the start of an update round (acts AND analyzes)
$\pi_\theta$	the live, currently-trainable policy
$A^{\text{rl}}_{q,n}$	the group-relative (GRPO) advantage for trajectory $n$
$\ell^{\text{skill}}, \ell^{\theta}$	teacher (skill-augmented) and student (ordinary) log-probabilities of the same sampled token
$\Delta$	the detached log-probability shift $\ell^{\text{skill}} - \ell^{\theta}$
$g = \sigma(\beta_{\text{opd}} \Delta)$	the confidence gate, a soft weight in $(0,1)$
$\beta_{\text{opd}}$	gate sharpness hyperparameter (5.0 in the paper)
$\lambda_{\text{opd}}$	OPD loss coefficient in the joint objective (0.01 in the paper)
$d_k(c)$	the token-context occupancy distribution induced by policy $\pi_k$
$w_k(c,v)$	the conditional expected gate for token $v$ at context $c$
$Z_k(c)$	the normalizer $\sum_v \pi_k(v\mid c) w_k(c,v)$
$r_k(\cdot\mid c)$	the skill-reweighted target distribution, $\pi_k(v\mid c)w_k(c,v)/Z_k(c)$
$Q_k(c,v)$	the expected task return from choosing $v$ at $c$ and following $\pi_k$ thereafter

PPO-Style Clipping: Why the RL Loss Isn’t Just “Advantage Times Log-Probability”

One more prerequisite worth spelling out before the method section, since it appears in Eq. (RL) and is easy to skim past: why does the RL loss use $\min(\rho A, \text{clip}(\rho, 1-\epsilon, 1+\epsilon) A)$ rather than simply $\rho A$ or even just $A \cdot \log \pi_\theta$? The probability ratio $\rho_{q,n,t,\ell}(\theta) = \exp(\ell^\theta_{q,n,t,\ell} - \ell^{\text{old}}_{q,n,t,\ell}) = \pi_\theta(a\mid c)/\pi_{\theta_{\text{old}}}(a\mid c)$ measures how much more (or less) likely the current, updating policy makes a given token relative to the frozen policy that actually generated it. Because $\theta$ changes across the inner optimization steps of one outer round (recall: rollouts and skills are all generated once, under a single frozen $\theta_{\text{old}}$, then multiple gradient steps are taken against that fixed batch of data), $\rho$ can, in principle, grow arbitrarily large or shrink toward zero as $\theta$ moves away from $\theta_{\text{old}}$ over these steps — an unclipped importance-weighted objective $\rho \cdot A$ would then allow a single large ratio to dominate the loss and induce a destructively large policy update from stale, off-policy-relative-to-the-new-$\theta$ data. The clipping in $\text{clip}(\rho, 1-\epsilon_{\text{clip}}, 1+\epsilon_{\text{clip}})$ caps how much credit (or blame) a single token’s ratio can receive: once $\rho$ moves outside the trust region $[1-\epsilon_{\text{clip}}, 1+\epsilon_{\text{clip}}]$ (0.2 in the paper’s setup, i.e., $\rho \in [0.8, 1.2]$), the objective’s sensitivity to further increases in $\rho$ is capped, and taking the $\min$ of the clipped and unclipped versions (for a positive advantage) ensures the objective doesn’t reward the policy for moving the ratio further in the already-favorable direction beyond the trust region — this is the standard PPO trust-region mechanism, applied here at the token level rather than the traditional per-timestep-of-a-single-decision level common in classical RL, since each token in the LLM’s generated action counts as one “decision” in this formulation.

A Closer Look at Why Outcome-Only RL Runs Out of Signal

It is worth dwelling on Proposition 2’s tied-reward scenario before diving into the method, because it is the sharpest, most concrete illustration of why SEED’s design is not just “add more supervision for the sake of it” but targets a specific, provable blind spot. Consider a rollout group where all $N$ sampled trajectories for a task happen to succeed (a very common situation once training has converged reasonably well — e.g., ALFWorld success rates above 80-90% mean most rollout groups are “all successes”). Then every $R(\tau_q^{(n)})$ in the group is identical, so $\mu_q$ equals every individual outcome, $R(\tau_q^{(n)}) - \mu_q = 0$ for every $n$, and the advantage $A^{\text{rl}}_{q,n}$ in Eq. (RL-Adv) is exactly zero for every token in every trajectory in the group. The clipped PPO-style surrogate loss built from this advantage therefore contributes exactly zero gradient — GRPO has nothing left to say about which of the (equally successful) trajectories took a more efficient path, or which individual actions within a successful trajectory were the genuinely good decisions versus incidental, wasteful detours that happened not to derail the outcome.

I verified this numerically (a small Python script, described in the Formula Derivations section below) and it is exactly as stark as the algebra suggests: with two trajectories both scoring $R=1.0$, the group mean is $1.0$, the deviation from mean is $0.0$ for both, and the resulting advantage vector is [0.0, 0.0] — a literal zero, not just a small number. SEED’s OPD loss, in the same scenario, has a gradient whose squared norm (under a natural inverse-policy-weighted metric) equals Var[w_k(c,·)] — the variance of the hindsight gate across candidate tokens at that context — which is nonzero as long as the hindsight skill actually discriminates between at least two candidate continuations. This is the mathematical content of “SEED provides dense credit even under sparse or tied rewards,” and it is the reason the paper’s ablation shows self-evolving OPD contributing a real, separate 4.8-point gain even on top of a system that already has hindsight-skill SFT.

Method: The SEED Training Pipeline

SEED runs two sequential training stages on top of a chosen backbone model (Qwen2.5-3B/7B-Instruct or Qwen3-1.7B-Instruct in the paper’s experiments), using the same set of weights throughout for both the acting and analyzing roles. Figure 2 (below, reproduced from the paper) gives the overall picture, and my own dataflow diagram unpacks the same pipeline with explicit named tensors.

flowchart TB
    subgraph Stage1["Stage 1: Hindsight Skill SFT (one-time)"]
        A1["Offline rollout collection:<br/>base policy x K0=8 trajectories/task"] --> A2["External analyzer (GLM-5.2)<br/>annotates hindsight skill per trajectory"]
        A2 --> A3["Format-validated (traj, skill) pairs"]
        A3 --> A4["Standard NLL fine-tuning:<br/>predict skill text from trajectory"]
        A4 --> A5["theta_sft checkpoint"]
    end

Figure A1 (self-drawn, dataflow overview, part 1 of 2): Stage 1 of the SEED pipeline, run once to give the model an initial trajectory-analysis capability before RL begins.

flowchart TB
    A5["theta_sft checkpoint"] --> B0
    subgraph Stage2["Stage 2: Self-Evolving On-Policy Distillation (per update)"]
        B0["Freeze current policy as theta_old"] --> B1["Sample N trajectories per task<br/>(theta_old acts)"]
        B1 --> B2["Compute group-relative advantage A_rl<br/>(GRPO term)"]
        B1 --> B3["Same theta_old, analyzer role:<br/>extract hindsight skill per trajectory"]
        B3 --> B4["Insert skill into history:<br/>skill-augmented context h~"]
        B4 --> B5["Re-score SAME sampled tokens:<br/>teacher log pi(a|h~) vs student log pi(a|h)"]
        B5 --> B6["Confidence gate g = sigmoid(beta * Delta)<br/>Delta = stopgrad(teacher - student)"]
        B2 --> B7["L_RL (clipped GRPO + KL)"]
        B6 --> B8["L_OPD (gate-weighted NLL on student branch)"]
        B7 --> B9["L_SEED = L_RL + lambda * L_OPD"]
        B8 --> B9
        B9 --> B10["Gradient step: theta updated"]
        B10 -->|"becomes next theta_old"| B0
    end

Figure A2 (self-drawn, dataflow overview, part 2 of 2): Stage 2 of the SEED pipeline, the actual RL loop, where the model plays both the actor role (collecting trajectories) and the analyzer role (annotating them), and the resulting hindsight-conditioned re-scoring drives an auxiliary distillation loss trained jointly with GRPO.

Formal Problem Setup

The paper formalizes the whole setting exactly as introduced in the Prerequisites section: the POMDP $(\mathcal{S}, \mathcal{A}, \mathcal{O}, \mathcal{T}, \Omega, \mathcal{R}, \gamma)$, with interaction history $h_t = (o_0, a_0, \dots, o_t)$, policy $a_t \sim \pi_\theta(\cdot \mid h_t)$, completed trajectory $\tau = \{(o_t, a_t, r_t)\}_{t=0}^{T-1}$, episode-level outcome $R(\tau)$, and the standard RL objective $J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)]$. The paper’s whole contribution can be read as: keep optimizing $J(\theta)$ via GRPO exactly as before, but add an auxiliary training-time-only signal derived from hindsight analysis of completed trajectories, that supplies gradient even in the tokens and situations where $J(\theta)$‘s trajectory-level gradient is uninformative.

Stage 1: Hindsight Skill Supervised Fine-Tuning, Step by Step

The goal of Stage 1 is narrow and mechanical: give the base policy the ability to read a completed trajectory and produce a useful natural-language hindsight skill, before that ability gets used as a live training signal in Stage 2.

1. Offline trajectory collection. For each of $M=180$ sampled training tasks $q_j$, run $K_0=8$ independent rollouts with the base (not-yet-fine-tuned) policy $\pi_{\theta_{\text{base}}}$: $\mathcal{B}_j = \{\tau_{j,k}\}_{k=1}^{K_0}$, $\tau_{j,k} \sim \pi_{\theta_{\text{base}}}(\cdot \mid q_j)$. Pooling over all tasks gives $\mathcal{B} = \bigcup_{j=1}^M \mathcal{B}_j$ — 1,440 completed trajectories total (180 tasks × 8 rollouts). Why collect these with the un-augmented base policy specifically: this stage’s SFT data must reflect ordinary (unmodified) agent-environment interaction, because the resulting checkpoint will later serve as the RL policy’s initialization — training it to analyze artificially-augmented trajectories would create a mismatch with the ordinary rollouts it will actually see during RL.
2. Hindsight skill annotation. Each completed trajectory $\tau$ is fed to an external analyzer $\mathcal{A}_{\text{ext}}$ (the paper uses GLM-5.2, temperature 0, max 4,096 tokens), which produces a skill annotation $s_\tau = \mathcal{A}_{\text{ext}}(\tau)$. For a successful trajectory, $s_\tau$ typically captures a reusable workflow (“first locate the target object, then take it to a cleaning station, clean it, then move it to the target location” — a real example from Table 4 of the paper’s Appendix). For a failed trajectory, $s_\tau$ captures a corrective or avoidance rule (“avoid moving an object to the target location before verifying both inventory and required object state”). Why use an external, more capable analyzer here rather than the base policy itself: at this early stage the base policy has no track record of producing coherent hindsight analysis, so bootstrapping with a stronger external model (rather than the model’s own untrained guesses) gives a cleaner initial training signal for the SFT step.
3. Format validation. Each annotation gets a validity indicator $v_\tau \in \{0,1\}$ (correctly formatted or not), and only valid pairs form the accepted SFT set: $\mathcal{D}_{\text{sft}} = \{(x_\tau, s_\tau) : \tau \in \mathcal{B}, v_\tau = 1\}$, where $x_\tau$ is the serialized trajectory-analysis input (the full trajectory transcript, formatted as a prompt asking the model to produce a hindsight skill).
4. Supervised fine-tuning. The base policy is fine-tuned with a standard autoregressive negative log-likelihood objective on the accepted pairs: $$\mathcal{L}_{\text{sft}}(\theta) = -\mathbb{E}_{(x_\tau,s_\tau)\sim\mathcal{D}_{\text{sft}}} \left[ \sum_{\ell=1}^{|s_\tau|} \log \pi_\theta(s_{\tau,\ell} \mid x_\tau, s_{\tau,<\ell}) \right]. \tag{SFT}$$ This is exactly the same objective used to train any instruction-following model — the only thing special is what it’s being taught to predict: given a full trajectory transcript as input, output a hindsight-skill string as the “response.” Three epochs of this training over $\mathcal{D}_{\text{sft}}$ produces $\theta_{\text{sft}}$, which initializes both the RL policy and the (now internal, no-longer-external) trajectory analyzer used in Stage 2.

What would go wrong without this stage (the “what if not” the depth clause requires): skipping Stage 1 and asking the un-fine-tuned base model to analyze its own trajectories directly during RL would likely produce low-quality, poorly formatted, or unhelpful skill annotations from the very start of training — and Table 2’s ablation confirms this is not a minor effect: removing hindsight-skill SFT drops ALFWorld’s average from 91.8 to 86.0, a real 5.8-point cost, showing that “equipping the actor and analyzer with an initial trajectory-analysis capability provides an important foundation for the subsequent self-evolving training process,” in the paper’s own words.

Stage 2: Self-Evolving On-Policy Distillation, Step by Step

This is where the actual training signal that gives the paper its name gets constructed. At the start of every policy-update round $k$, the current policy is frozen as $\pi_{\theta_{\text{old}}}$, and this single frozen snapshot plays two roles simultaneously: it collects the new trajectories (the “actor”), and it analyzes those same completed trajectories into hindsight skills (the “analyzer”). This dual-role sharing is the mechanical basis of the “self-evolving” property.

1. On-policy trajectory sampling. For each task prompt $q$ in the current batch, sample a group of $N=8$ trajectories from the frozen policy: $\mathcal{G}_q = \{\tau_q^{(1)}, \dots, \tau_q^{(N)}\}$, $\tau_q^{(n)} \sim \pi_{\theta_{\text{old}}}(\cdot \mid q)$.
2. Group-relative advantage (the RL half). Exactly as in vanilla GRPO: compute $\mu_q, \sigma_q$ from the group’s outcomes and derive $A^{\text{rl}}_{q,n}$ per Eq. (RL-Adv) above. Nothing about this half of the objective is new to SEED.
3. On-policy hindsight skill generation (the analyzer half). For each completed trajectory $\tau_q^{(n)}$, construct its trajectory-analysis input $x_{\tau_q^{(n)}}$ and feed it to the analyzer role of the same checkpoint: $s_q^{(n)} = \mathcal{A}_{\theta_{\text{old}}}(x_{\tau_q^{(n)}})$. Crucially, $\mathcal{A}_{\theta_{\text{old}}}$ uses the exact same parameters $\theta_{\text{old}}$ that just produced $\tau_q^{(n)}$ — there is no separate analyzer network, and no analyzer checkpoint lag.
4. Skill-augmented re-scoring. Let $H$ be a deterministic function that splices a hindsight skill into an interaction history without touching the already-sampled action tokens (i.e., it only modifies the context, not the sequence being scored). At timestep $t$, define the skill-augmented history $\tilde h_{q,n,t} = H(h_{q,n,t}, s_q^{(n)})$. The same current, trainable policy $\pi_\theta$ then computes two log-probabilities for the exact same already-sampled action tokens $a_{q,n,t} = (a_{q,n,t,1}, \dots, a_{q,n,t,L_{q,n,t}})$: $$\ell^{\text{skill}}_{q,n,t,\ell} = \log \pi_\theta(a_{q,n,t,\ell} \mid \tilde h_{q,n,t}, a_{q,n,t,<\ell}) \qquad \text{(teacher branch)} \tag{Teacher}$$ $$\ell^{\theta}_{q,n,t,\ell} = \log \pi_\theta(a_{q,n,t,\ell} \mid h_{q,n,t}, a_{q,n,t,<\ell}) \qquad \text{(student branch)}. \tag{Student}$$ Why re-score the same tokens rather than let each branch generate independently: this is the entire trick that lets both “teacher” and “student” be the same weights without collapsing into a trivial identity. Because the tokens $a_{q,n,t,\ell}$ are fixed (already sampled during rollout, under the ordinary/student context), asking “how much more or less likely would this exact token have been if the hindsight skill had been visible” is a well-posed, meaningful counterfactual question — even though both log-probabilities come from the identical parameter vector $\theta$.
5. The confidence gate. Define the detached (stop-gradient) log-probability shift $\Delta_{q,n,t,\ell} = \text{sg}[\ell^{\text{skill}}_{q,n,t,\ell} - \ell^{\theta}_{q,n,t,\ell}]$, and the gate $g_{q,n,t,\ell} = \sigma(\beta_{\text{opd}} \Delta_{q,n,t,\ell})$, exactly as described in Prerequisites. Both the teacher log-probability and the gate are detached — gradients only ever flow through the student branch $\ell^\theta$.
6. The OPD loss (Eq. 1): $$\mathcal{L}_{\text{opd}}(\theta) = \mathbb{E}_{q,n,t,\ell}\Big[ m_{q,n,t,\ell} \cdot g_{q,n,t,\ell} \cdot \text{sg}\big[\ell^{\text{skill}}_{q,n,t,\ell} - \ell^{\theta}_{q,n,t,\ell}\big] \Big]. \tag{1}$$ Because the gate and the teacher log-probability are both detached, differentiating gives $$\nabla_\theta \mathcal{L}_{\text{opd}} = -\mathbb{E}_{q,n,t,\ell}\Big[ m_{q,n,t,\ell} \cdot g_{q,n,t,\ell} \cdot \nabla_\theta \ell^{\theta}_{q,n,t,\ell} \Big],$$ which is exactly a gate-weighted negative-log-likelihood gradient: minimizing this loss increases the ordinary (student-context) log-probability of each sampled token, weighted by how strongly the hindsight skill endorsed that token. Tokens the skill strongly endorses ($g \approx 1$) get pushed up hard; tokens the skill disapproves of ($g \approx 0$) barely move.
7. Joint objective. The RL loss follows standard clipped-PPO-with-KL form using the token-level probability ratio $\rho_{q,n,t,\ell}(\theta) = \exp(\ell^\theta_{q,n,t,\ell} - \ell^{\text{old}}_{q,n,t,\ell})$: $$\mathcal{L}_{\text{rl}}(\theta) = -\mathbb{E}_{q,n,t,\ell}\Big[\min\big(\rho_{q,n,t,\ell}(\theta) A^{\text{rl}}_{q,n,t,\ell},\ \text{clip}(\rho_{q,n,t,\ell}(\theta), 1-\epsilon_{\text{clip}}, 1+\epsilon_{\text{clip}}) A^{\text{rl}}_{q,n,t,\ell}\big)\Big] + \beta_{\text{KL}} D_{\text{KL}}, \tag{RL}$$ and the final training objective is $$\mathcal{L}_{\text{SEED}}(\theta) = \mathcal{L}_{\text{rl}}(\theta) + \lambda_{\text{opd}} \mathcal{L}_{\text{opd}}(\theta), \tag{SEED}$$ with $\lambda_{\text{opd}} = 0.01$ in the paper’s experiments — a deliberately small coefficient, since the auxiliary loss’s job is to nudge the policy toward hindsight-endorsed tokens, not to dominate the outcome-driven RL signal. After the gradient step, the updated $\pi_\theta$ becomes $\pi_{\theta_{\text{old}}}$ for the next iteration, closing the self-evolving loop: the actor that generates the next round’s trajectories, and the analyzer that will interpret them, are both the version of the model that just learned from this round’s hindsight.
8. Deployment. At inference time the agent acts only from the ordinary history, $a_t \sim \pi_\theta(\cdot \mid h_t)$ — no analyzer, no skill bank, no augmented prompt. All the hindsight-skill machinery is training-time-only scaffolding.

A Closer Look at the Format-Validation Step and Why It Matters

Step 3 of Stage 1 (format validation, producing the validity indicator $v_\tau$) is easy to skim past as a minor data-cleaning detail, but it deserves a moment’s attention because of what it implies about the quality-control burden this pipeline carries. Because Stage 1’s SFT data comes from an external analyzer (GLM-5.2) generating free-form natural-language text, there is no guarantee every generated skill is well-formed, appropriately scoped, or even non-empty — the external model could produce overly long, truncated (if it hits the 4,096-token cap), malformed, or off-topic text for some fraction of the 1,440 offline trajectories. The validity indicator $v_\tau \in \{0, 1\}$ filters these out before they enter $\mathcal{D}_{\text{sft}}$, but the paper does not report what fraction of the 1,440 generated annotations actually failed this check, nor what the validation criteria specifically are (beyond “correctly formatted”) — a filtering step whose acceptance rate is unreported is a small but real reproducibility gap, since a very low acceptance rate would mean a much smaller effective SFT dataset than the nominal 1,440-trajectory pool suggests, while a very high acceptance rate would suggest the check is largely a formality. Neither number is given.

Stage 2’s on-policy skill generation (step 3 of the per-round loop, $s_q^{(n)} = \mathcal{A}_{\theta_{\text{old}}}(x_{\tau_q^{(n)}})$) does not appear to re-apply this same validity check — the paper’s Algorithm 1 shows the skill being generated and immediately used to construct the skill-augmented context $\tilde h_{q,n,t}$ (line 13), with no visible filtering step analogous to Stage 1’s $v_\tau$. This is a reasonable design choice in one sense (Stage 2’s OPD loss is inherently self-correcting: if a badly-formed skill produces no meaningful log-probability shift, the resulting gate $g \approx \sigma(0) = 0.5$ contributes only a mild, near-neutral nudge, rather than actively corrupting training), but it does mean the paper offers no direct evidence about how often Stage 2’s on-policy analyzer (which, remember, started from the Stage-1 SFT checkpoint and continues to evolve through RL) produces malformed or degenerate skills during actual training, versus how reliably well-formed its output remains as it moves further from its Stage-1-SFT initialization over 150 update rounds.

The Complete Algorithm, Line by Line

The paper’s Algorithm 1 (Appendix B.3) is reproduced below with added commentary on what each block accomplishes and why it is ordered this way.

Algorithm 1: SEED (Self-Evolving On-Policy Distillation)
Require: SFT-initialized policy π_θsft, task set Q, context function H,
         group size N, gate sharpness β_opd, KL coeff β_KL,
         OPD coefficient λ_opd, clip ε_clip, learning rate η
Ensure:  Trained policy π_θ

 1: θ ← θ_sft                              // start from Stage-1 checkpoint
 2: for each policy update do
 3:     θ_old ← θ                          // FREEZE current policy: this single
                                            // snapshot will act AND analyze below
 4:     Sample a task batch B ⊂ Q
 5:     // ---- On-policy experience + synchronized skill analysis ----
 6:     for each task q ∈ B do
 7:         Sample G_q = {τ_q^(n)}_{n=1..N} with τ_q^(n) ~ π_θold(·|q)
                                            // actor role of θ_old
 8:         Compute μ_q, σ_q from {R(τ_q^(n))}
 9:         for each trajectory τ_q^(n) in G_q do
10:             A_rl[q,n] ← (R(τ_q^(n)) - μ_q) / (σ_q + ε)
                                            // vanilla GRPO advantage
11:             s_q^(n) ← A_θold(x_τq(n))  // analyzer role of θ_old --
                                            // SAME checkpoint, different "hat"
12:             for each action step t in τ_q^(n) do
13:                 h~[q,n,t] ← H(h[q,n,t], s_q^(n))
                                            // splice skill into history
14:                 cache ℓ_old[q,n,t,ℓ]    // importance-ratio reference,
                                            // for ALL valid sampled tokens
15:             end for
16:         end for
17:     end for
18:     // ---- Paired contextual re-scoring + joint optimization ----
19:     for each minibatch of valid sampled tokens do
20:         evaluate ℓ_skill[q,n,t,ℓ] and ℓ_θ[q,n,t,ℓ] using CURRENT π_θ
                                            // teacher branch (skill context)
                                            // student branch (ordinary context)
21:         Δ[q,n,t,ℓ] ← stopgrad(ℓ_skill - ℓ_θ)
22:         g[q,n,t,ℓ] ← sigmoid(β_opd · Δ) // confidence gate, ALSO detached
23:         ρ[q,n,t,ℓ](θ) ← exp(ℓ_θ - ℓ_old)
                                            // token-level PPO importance ratio
24:         compute clipped GRPO loss L_rl from ρ, A_rl, ε_clip
25:         L_opd ← E[ mask · g · stopgrad(ℓ_skill - ℓ_θ) ]
                                            // gradients flow ONLY through ℓ_θ
26:         L_SEED ← L_rl + λ_opd · L_opd
27:         θ ← θ - η ∇_θ L_SEED
28:     end for
29: end for                                 // θ_old is refreshed to the NEW θ
                                             // next iteration -> self-evolving loop

Two structural points worth flagging explicitly. First, lines 3-17 all happen under the frozen $\theta_{\text{old}}$ — this is the “data generation” phase, and it is where both the trajectories and the skills used to interpret them get produced by the same, momentarily-fixed model. Second, lines 19-28 are where the trainable $\theta$ diverges from $\theta_{\text{old}}$ across the inner optimization steps of one outer round — the re-scoring in line 20 uses the live, updating $\pi_\theta$, while line 23’s importance ratio still references the frozen $\ell_{\text{old}}$ from data-generation time. This two-phase structure (freeze-and-collect, then optimize) is exactly standard PPO/GRPO practice, extended here to also freeze-and-collect the hindsight skills, not just the trajectories.

A Complete Worked Numeric Trace of One SEED Update Round

Before diving into the formal propositions, it helps to see the entire mechanism traced through on one small, fully concrete example — not the tied-reward edge case discussed above, but the “normal” operating regime where GRPO already has some signal, to show how OPD’s contribution differs in kind, not just in degree.

Setup. A rollout group of 2 trajectories for one task: trajectory 1 succeeds ($R=1$), taking three actions [good, good, good]; trajectory 2 fails ($R=0$), taking three actions [good, bad, bad] — note that trajectory 2’s first action was actually fine; only the second and third actions were the real mistakes. This is exactly the kind of situation the paper’s introduction describes in prose (“a failed trajectory may contain useful partial behaviors but fail because of a few local mistakes”).

GRPO’s view. Group mean $\mu = 0.5$, std $\sigma = 0.5$, giving $A^{\text{rl}} = [+1.0, -1.0]$ (up to the $\epsilon$ term). Trajectory 1’s advantage ($+1.0$) is broadcast to all three of its steps; trajectory 2’s advantage ($-1.0$) is broadcast to all three of its steps too — including step 1, which was actually a perfectly reasonable action. GRPO has no way to single out that trajectory 2’s problem was specifically steps 2-3, not step 1.

SEED’s additional view, via the hindsight skill for trajectory 2. Suppose the analyzer, examining the complete failed trajectory, extracts a skill like “the first action was fine, but repeating the second choice after an initial mistake compounds the failure — avoid choosing the same wrong action twice in a row.” I constructed toy student (ordinary-context) and teacher (skill-augmented-context) log-probabilities for the three sampled actions in trajectory 2, chosen to reflect this skill’s likely effect (verified via direct sigmoid computation, not hand-waved):

Step	Action	Student log-prob $\ell^\theta$	Teacher log-prob $\ell^{\text{skill}}$	$\Delta$	Gate $g=\sigma(5\Delta)$
1	good	$-0.350$	$-0.330$	$+0.020$	$0.5250$
2	bad	$-0.900$	$-1.400$	$-0.500$	$0.0759$
3	bad	$-1.100$	$-2.600$	$-1.500$	$0.0006$

Reading this table. Step 1’s gate ($0.525$) sits almost exactly at the neutral midpoint $0.5$ — the hindsight skill has essentially no opinion about this action, consistent with it being a reasonable choice the skill doesn’t need to correct. Steps 2 and 3’s gates are low ($0.076$ and $0.0006$ respectively) — the skill actively discourages both sampled bad actions, and discourages the third step’s action far more strongly than the second, exactly matching the skill’s own content (“avoid repeating the mistake” implies the second occurrence of the bad choice is worse than the first). Because the OPD gradient (per Eq. 10-style reasoning) pushes the ordinary policy’s log-probability up in proportion to the gate, step 1 gets pushed up almost as if untouched by the auxiliary loss (gate $\approx 0.5$, a near-neutral weighting), while steps 2 and 3 get their (negative, discouraging) GRPO signal essentially undiminished by the auxiliary term, since a low gate means the OPD loss contributes very little upward pressure to counteract GRPO’s downward pressure on those specific tokens. In other words: GRPO alone would suppress all three actions in trajectory 2 equally; adding SEED’s gated OPD term leaves the suppression of steps 2-3 essentially intact while partially offsetting the suppression of step 1 — exactly the fine-grained, within-trajectory correction that a single scalar advantage cannot provide. This is a small, illustrative, hand-constructed example (not reproduced from the paper’s own logs), but it demonstrates concretely, with real numbers, the qualitative mechanism the paper describes only in prose.

Formula Derivations: What the Three Propositions Actually Prove

The paper’s Appendix A gives three formal propositions that mirror the three design requirements stated in the introduction (on-policy, dense, self-evolving). I re-derive and numerically verify each one below, because the paper’s proofs are correct but terse, and because “the gradient is provably nonzero under tied rewards” is exactly the kind of claim that deserves independent numeric verification rather than being taken on faith.

Proposition 1: The OPD Update Is Occupancy-Matched KL-Distillation

Setup. Fix an outer iteration $k$, with behavior policy $\pi_k = \pi_{\theta_k}$ and analyzer $\mathcal{A}_k$. Let $c$ denote an ordinary context at a valid action-token position, $d_k(c)$ the token-context occupancy induced by $\pi_k$ and the environment, and $Y \sim \pi_k(\cdot \mid c)$ the realized sampled token. After the trajectory completes, the analyzer produces skill $S = \mathcal{A}_k(x_\tau)$, and the detached gate is $g_k(c, v, S) = \sigma(\beta_{\text{opd}}[\log \pi_k(v \mid H(c,S)) - \log \pi_k(v \mid c)])$. Define the conditional expected gate, marginalizing over the future trajectory and analyzer randomness:

$$w_k(c,v) = \mathbb{E}[g_k(C,Y,S) \mid C=c, Y=v]. \tag{3}$$

Proposition 1. Define $Z_k(c) = \sum_{v} \pi_k(v\mid c) w_k(c,v)$ and $r_k(v \mid c) = \pi_k(v\mid c)w_k(c,v)/Z_k(c)$. Then $r_k(\cdot\mid c)$ is a valid probability distribution, and

$$\nabla_\theta \mathcal{L}_{\text{opd},k}(\theta)\Big|_{\theta=\theta_k} = \mathbb{E}_{c\sim d_k}\Big[ Z_k(c) \cdot \nabla_\theta D_{\text{KL}}(r_k(\cdot\mid c) \,\|\, \pi_\theta(\cdot\mid c))\Big|_{\theta=\theta_k} \Big]. \tag{5}$$

Derivation, step by step. At $\theta = \theta_k$, expand the OPD gradient by conditioning first on $C=c$, then on $Y=v$: because both the teacher log-probability and the gate are detached, only the student log-probability contributes,

$$\nabla_\theta \mathcal{L}_{\text{opd},k}\Big|_{\theta_k} = -\mathbb{E}_{c\sim d_k}\Big[\sum_v \pi_k(v\mid c)\, w_k(c,v)\, \nabla_\theta \log \pi_\theta(v\mid c)\Big|_{\theta_k}\Big].$$

Substituting $\pi_k(v\mid c) w_k(c,v) = Z_k(c) r_k(v\mid c)$ (the definition of $r_k$) gives

$$= -\mathbb{E}_{c\sim d_k}\Big[ Z_k(c) \sum_v r_k(v\mid c) \nabla_\theta \log \pi_\theta(v\mid c)\Big|_{\theta_k}\Big],$$

and the sum $\sum_v r_k(v\mid c) \nabla_\theta \log \pi_\theta(v\mid c)$ is exactly $-\nabla_\theta D_{\text{KL}}(r_k(\cdot\mid c)\,\|\,\pi_\theta(\cdot\mid c))$ up to the sign convention (the entropy term of the detached target $r_k$ has zero gradient with respect to $\theta$, so differentiating cross-entropy and differentiating KL divergence give the same result here). This proves Eq. (5).

What this means, in plain language. The OPD update is not “imitate whatever the skill-augmented teacher says” — it is “distill toward a re-weighted version of the current policy’s own distribution, where re-weighting is governed by how strongly the hindsight skill endorses each candidate token relative to the average.” The factor $d_k(c)$ (occupancy) ensures the update only ever touches contexts the current policy actually visits — this is the “on-policy” half. The factor $w_k(c,v)$ (skill-selectivity) ensures that, within each such context, the update discriminates between candidate tokens by how strongly the hindsight skill supports them.

Numerical verification (constructed by this reviewer). I built a toy 4-token vocabulary at a fixed context, with $\pi_k = [0.10, 0.35, 0.40, 0.15]$ and gate values $w_k = [0.20, 0.35, 0.85, 0.70]$ (token 2 most strongly endorsed, token 0 least). Computing $Z_k = \sum_v \pi_k(v) w_k(v) = 0.5875$ and $r_k = \pi_k \cdot w_k / Z_k = [0.034, 0.209, 0.579, 0.179]$ (verified to sum to 1.0), I confirmed the ratio-monotonicity claim implicit in the derivation — $r_k(v)/\pi_k(v)$ is strictly increasing in $w_k(v)$ — holds exactly: sorting tokens by $r_k/\pi_k$ gives the identical order as sorting by $w_k$ directly ([2,3,1,0] in both cases). I then computed the OPD loss’s gradient with respect to the logits two independent ways: (a) via finite-difference numerical differentiation of the loss function directly, and (b) via the closed-form expression implied by the softmax-cross-entropy identity, $\nabla_{z(v)}\mathcal{L}_{\text{opd}} = \pi_\theta(v)Z_k - \pi_k(v)w_k(v)$. Both methods agreed to 8 decimal places (max absolute difference $\approx 0$), confirming: tokens with $w_k(v) > Z_k$ (above-average endorsement — tokens 2 and 3 in my example) get a negative gradient (probability mass pushed up), while tokens with $w_k(v) < Z_k$ (below-average endorsement — tokens 0 and 1) get a positive gradient (probability mass pushed down) — exactly the “signed relative credit despite all-nonnegative gate values” behavior the paper describes after Eq. 10-11.

The Value-Alignment Condition (Eq. 8): What Makes OPD Actually Helpful, Not Just Well-Defined

Proposition 1 shows the OPD update is well-defined as occupancy-matched distillation, but it does not by itself show the update helps. The paper’s Eq. 8 supplies the missing condition. Let $Q_k(c,v)$ be the expected task return from choosing token $v$ at context $c$ and following $\pi_k$ thereafter. Then

$$\mathbb{E}_{v\sim r_k(\cdot\mid c)}[Q_k(c,v)] - \mathbb{E}_{v\sim \pi_k(\cdot\mid c)}[Q_k(c,v)] = \frac{\text{Cov}_{v\sim\pi_k(\cdot\mid c)}(Q_k(c,v), w_k(c,v))}{Z_k(c)}. \tag{8}$$

Reading this equation: the left side is “how much better is the reweighted target’s expected value than the raw policy’s expected value” — exactly the quantity you want to be positive if the OPD update is going to help rather than hurt. The right side says this quantity equals a covariance, divided by the (always positive) normalizer $Z_k(c)$: whenever the hindsight gate $w_k(c,v)$ is positively correlated (across candidate tokens, weighted by the current policy) with the true action value $Q_k(c,v)$, the covariance is positive, and the reweighted target genuinely has higher expected value than the unweighted policy. This is the condition the whole method’s practical success rests on, and it is explicitly not automatic — it depends on the hindsight skills the analyzer produces actually being correlated with genuinely good decisions, not just plausible-sounding but ultimately uninformative text.

I verified the identity numerically using the same 4-token toy setup, adding a toy “value” vector $Q_k = [0.10, 0.30, 0.90, 0.60]$ (chosen so the highest-value token, index 2, also has the highest gate, index 2 again — a deliberately favorable case). Direct computation gives $\mathbb{E}_{r_k}[Q] - \mathbb{E}_{\pi_k}[Q] = 0.1290$, and the covariance-based right-hand side, $\text{Cov}_{\pi_k}(Q,w)/Z_k$, gives the identical $0.1290$ — confirming the algebraic identity exactly, and illustrating concretely why aligning $w_k$ with $Q_k$ (i.e., having the analyzer’s endorsements track true decision quality) is the load-bearing assumption behind every empirical result in the paper.

Proposition 2: Dense Signal Survives Even When Every Trajectory Ties

This is the proposition I dwelt on qualitatively above; here is its exact statement and my numeric confirmation. Setup. Consider a rollout group where every trajectory has the same outcome, so $A^{\text{rl}}_{q,n,t,\ell} = 0$ for every valid token (the RL gradient is provably zero). Parameterize the ordinary student distribution at a fixed context $c$ as $p_z(\cdot\mid c) = \text{softmax}(z(c))$, and define the conditional expected OPD loss up to detached constants:

$$\mathcal{L}_{\text{opd},k,c}(z) = -\sum_v \pi_k(v\mid c)\, w_k(c,v)\, \log p_z(v\mid c). \tag{9}$$

Result. At $\theta = \theta_k$ (i.e., $p_z(\cdot\mid c) = \pi_k(\cdot\mid c)$),

$$\frac{\partial \mathcal{L}_{\text{opd},k,c}}{\partial z(c,v)} = \pi_k(v\mid c)\big[Z_k(c) - w_k(c,v)\big], \tag{10}$$

and, using the inverse-policy-weighted squared norm $\|a\|^2_{\pi_k^{-1}} = \sum_v a(v)^2/\pi_k(v\mid c)$,

$$\big\|\nabla_{z(c)}\mathcal{L}_{\text{opd},k,c}\big\|^2_{\pi_k^{-1}} = \text{Var}_{v\sim\pi_k(\cdot\mid c)}\big[w_k(c,v)\big]. \tag{11}$$

Hence, the OPD gradient is nonzero if and only if the expected hindsight gate is non-constant over candidate tokens with positive current-policy probability — i.e., exactly when the hindsight skill actually discriminates among the plausible continuations at this context.

Numerical verification (my own toy example). Two trajectories, both with reward $R=1.0$. Group mean $\mu = 1.0$, group std $\sigma = 0.0$, so $A^{\text{rl}} = (R-\mu)/(\sigma+\epsilon) = [0.0, 0.0]$ — a literal, exact zero, confirming the RL gradient contributes nothing. Meanwhile, reusing the same $\pi_k, w_k$ toy vectors from Proposition 1’s verification, $\text{Var}_{\pi_k}[w_k] = 0.0642$ — nonzero, as claimed. I further verified Eq. 11’s exact identity numerically: computing $\|\nabla_{z(c)}\mathcal{L}_{\text{opd}}\|^2_{\pi_k^{-1}} = \sum_v (\pi_k(v)[Z_k - w_k(v)])^2/\pi_k(v)$ directly from the closed-form gradient in Eq. 10, I got $0.0642$, matching $\text{Var}_{\pi_k}[w_k]$ exactly. This is the concrete mechanism by which SEED distinguishes a locally useful decision from a locally harmful one inside a group of trajectories that all end in success — a regime where GRPO alone genuinely has nothing left to say.

Proposition 3: Refreshing the Analyzer Bounds Supervision Staleness

Setup. For a completed token sample $x = (\tau, c, v)$ drawn from the current trajectory distribution $\mu_k$, let $s_r(x) = \mathcal{A}_r(x_\tau)$ be the skill produced by an analyzer checkpoint from some earlier iteration $r$. Holding the current policy $\pi_k$ fixed for comparison, define $\Delta_{k,r}(x) = \log\pi_k(v\mid H(c,s_r(x))) - \log\pi_k(v\mid c)$ and $g_{k,r}(x) = \sigma(\beta_{\text{opd}}\Delta_{k,r}(x))$ — i.e., what the gate would have been if an older analyzer $\mathcal{A}_r$ (instead of the current one $\mathcal{A}_k$) had produced the skill. Define the resulting expected auxiliary gradient $U_k(\mathcal{A}_r;\theta) = -\mathbb{E}_{x\sim\mu_k}[g_{k,r}(x)\nabla_\theta \log\pi_\theta(v\mid c)]$.

Result (Eq. 14). Assuming $\|\nabla_\theta \log\pi_\theta(v\mid c)\|_2 \le G$,

$$\big\|U_k(\mathcal{A}_k;\theta) - U_k(\mathcal{A}_j;\theta)\big\|_2 \le G\, \mathbb{E}_{x\sim\mu_k}\big[|g_{k,k}(x) - g_{k,j}(x)|\big] \le \frac{\beta_{\text{opd}} G}{4}\, \mathbb{E}_{x\sim\mu_k}\big[|\Delta_{k,k}(x) - \Delta_{k,j}(x)|\big]. \tag{14}$$

Where the second inequality comes from: the map $z \mapsto \sigma(\beta_{\text{opd}} z)$ is $\beta_{\text{opd}}/4$-Lipschitz (because $\sup_z \sigma'(z) = 1/4$), applied pointwise. Crucially, using the analyzer instantiated from the current checkpoint (i.e., $j=k$) sets this entire discrepancy to exactly zero at the rollout-and-analysis stage — this is precisely what SEED does by construction, so the bound formalizes why self-evolving synchronization (as opposed to a periodically-refreshed or entirely static analyzer) removes a real source of gradient bias, rather than being a mere implementation convenience.

Numerical illustration (my own worked example). Consider a single sampled token where the ordinary (student) log-probability is $\log\pi_k(v\mid c) = -1.20$. A current analyzer’s skill (produced by $\mathcal{A}_k$, matching the actor’s own current failure modes) endorses this token strongly, giving a skill-augmented log-probability of $-0.40$, so $\Delta_{k,k} = 0.80$. A stale analyzer, from an earlier training checkpoint that hasn’t yet learned to recognize this particular failure/success pattern, endorses the same token more weakly, giving $-0.90$, so $\Delta_{k,j} = 0.30$. With $\beta_{\text{opd}}=5.0$: $g_{k,k} = \sigma(5.0 \times 0.80) = 0.9820$, $g_{k,j} = \sigma(5.0\times 0.30) = 0.8176$, a gate discrepancy of $|g_{k,k}-g_{k,j}| = 0.1644$. The Lipschitz-based bound gives $(\beta_{\text{opd}}/4)\times|\Delta_{k,k}-\Delta_{k,j}| = 1.25 \times 0.50 = 0.625$ — comfortably above the actual $0.1644$ discrepancy, confirming the bound holds (as it must, being a proven inequality), while also showing the bound is not tight in this instance — a looseness the paper does not itself discuss (see Critical Assessment). This example is purely illustrative: in the actual SEED algorithm, $j$ is never used — the analyzer is always the current checkpoint, by construction, exactly so this term never has to be paid.

Connecting the Confidence Gate to Classical Temperature-Scaled Distillation

Readers coming from a classical knowledge-distillation background may find it useful to see how SEED’s confidence gate relates to the more familiar temperature-scaled soft-label distillation from Hinton et al.’s original knowledge distillation paper. In classical distillation, a teacher’s output distribution is softened with a temperature $T$: $p_{\text{teacher}}(v) \propto \exp(z_{\text{teacher}}(v)/T)$, and the student is trained to match this softened distribution via a KL or cross-entropy loss over the entire vocabulary, at every generated position, without any notion of “confidence” in the teacher’s output — every teacher probability, however uncertain, contributes as a soft target with equal footing.

SEED departs from this classical picture in three specific ways, each worth naming explicitly. First, rather than distilling the entire teacher distribution over the vocabulary, SEED only ever supervises the single already-sampled token $a_{q,n,t,\ell}$ — this is a sampled-token distillation objective (following prior on-policy distillation work), not a full-distribution KL match. Second, rather than using a fixed temperature to soften an already-confident teacher, SEED’s gate $g = \sigma(\beta_{\text{opd}}\Delta)$ operates on the difference between two log-probabilities (teacher context vs. student context) of the same underlying model, not on the teacher’s own output sharpness. This is a fundamentally different quantity: it measures how much a specific piece of side-information (the hindsight skill) shifted the model’s opinion about one specific token, not how uncertain the model’s overall output distribution happens to be. Third, and most importantly, the direction of information flow is inverted relative to classical distillation: in classical KD, a fixed, already-trained teacher imparts knowledge to an untrained (or differently-trained) student; in SEED, both “teacher” and “student” are the same evolving parameters, and the “knowledge” being transferred is not pre-existing expertise but a counterfactual self-observation — what the model itself would have predicted, had it been given a piece of information (the hindsight skill) that it generated about its own past behavior. This closes a loop that classical distillation, by design, never has: the model is teaching itself using commentary about its own trajectories, not learning from an external source of ground truth.

This distinction also clarifies why $\beta_{\text{opd}}$ plays a different role than temperature $T$ in classical distillation. Temperature $T$ controls how soft or sharp the teacher’s full distribution appears to the student, uniformly across all tokens and positions. $\beta_{\text{opd}}$ instead controls how sharply the gate itself discriminates between “the skill helped” and “the skill didn’t help,” for one specific already-realized token — it is a meta-parameter over the confidence signal, not over the distribution being matched. Practically, this means tuning $\beta_{\text{opd}}$ answers a different question than tuning a distillation temperature would: it is asking “how quickly should small log-probability shifts saturate into strong endorsement or rejection,” not “how much should I smooth out the teacher’s peakiness.”

A Direct Comparison Table: SEED vs. the Baseline Landscape

To make the taxonomy of “what exactly is being compared against what” concrete, here is a condensed version of the paper’s baseline categories, annotated with the specific mechanism each one is missing relative to SEED.

Method	Uses skills at train time?	Uses skills at test time?	Dense (token-level) credit?	Analyzer synchronized w/ current policy?
Vanilla	No	No	No (no post-training)	N/A
Skill-Prompt*	No (frozen weights)	Yes (prompt)	N/A	N/A
GRPO	No	No	No (trajectory-level only)	N/A
Skill-GRPO	Yes (in context, affects exploration)	No (removed at test)	No (still trajectory-level advantage)	N/A (skill is retrieved, not analyzer-produced)
Skill-GRPO*	Yes	Yes (prompt kept)	No	N/A
OPSD	Yes (privileged context)	No	Yes	Not skill-based; generic privileged context
GRPO+OPSD	Yes	No	Yes	Not skill-based
Skill-SD	Yes (retrieved skill)	No	Yes	No (retrieved from a library, not self-analyzed)
RLSD	Yes (privileged self-teacher)	No	Partial (modulates GRPO magnitude, not sign)	Decays toward vanilla GRPO over training
SDAR	Yes (privileged self-teacher, gated)	No	Yes (separately gated distillation term)	Gate depends on student uncertainty, not skill freshness per se
SEED	Yes (self-generated, on-policy)	No	Yes (Eq. 1, confidence-gated)	Yes (same checkpoint plays both roles, every round)

The row that most sharply isolates SEED’s specific contribution is the comparison against SDAR, RLSD, Skill-SD, and OPSD: all four already use some form of privileged self-teacher and dense token-level distillation, yet SEED still beats the strongest of them (SDAR: 84.4 ALFWorld average) by 7.4 points at the 3B scale, by 10.2 points at 7B, and by a striking 38.1 points at the smaller Qwen3-1.7B scale — where a smaller model’s own trajectories and failure modes shift more over the course of training, the value of a genuinely self-evolving (rather than merely gated or decaying) analyzer becomes correspondingly larger.

Related Work in Context: How SEED’s Baselines Themselves Differ

Because the paper compares against five distinct self-distillation baselines (OPSD, GRPO+OPSD, Skill-SD, RLSD, SDAR), and it is easy to lose track of what specifically differentiates each one, it is worth walking through their mechanisms one at a time, in the order the paper introduces them, before returning to how SEED differs from all five.

OPSD (Zhao et al., 2026). The most generic of the five: student and teacher branches share the same underlying model, conditioned on different contexts, where the teacher additionally observes some privileged information available only during training (not necessarily a natural-language skill — could be any augmented context). The teacher re-scores the student’s sampled tokens, producing dense token-level targets via distribution matching, with teacher-side outputs detached. This is architecturally the closest ancestor of SEED’s re-scoring mechanism, but OPSD does not specify what the privileged context should be, nor does it involve any notion of hindsight analysis of the student’s own trajectories — the privileged information in generic OPSD could just as easily be a longer reasoning trace, a ground-truth label, or any other side-channel. Notably, OPSD alone (without combining it with GRPO) performs quite poorly on ALFWorld (28.1 average) — comparable to Vanilla and far below GRPO’s 75.0 — showing that dense token-level distillation alone, without an outcome-driven RL objective underneath it, is not sufficient; this is why the paper also tests GRPO+OPSD (jointly optimizing both objectives, 81.2 average), which serves as the controlled comparison for “is a straightforward combination of outcome-based RL and generic self-distillation enough,” and the answer, given SEED’s 91.8, is no.

Skill-SD (Wang et al., 2026). Adapts self-distillation specifically to the skill-based setting: completed experience is represented as natural-language skills, and a retrieved skill (from some pre-built or growing library, not necessarily generated by the current policy) is supplied only to the teacher branch, while the student trains from the ordinary context. This is architecturally close to SEED in using natural-language skills specifically, but the skill is retrieved, not self-generated by analyzing the current trajectory — meaning Skill-SD’s skill source can, in principle, be entirely independent of what the student model itself is currently doing.

RLSD (Yang et al., 2026a). Takes a different mechanical approach: rather than adding an independent distillation loss term, RLSD converts the teacher-student log-probability gap into a bounded coefficient that modulates the magnitude of each token’s existing GRPO update, while the sign of the update remains fully determined by the environment-derived advantage. This is an important structural distinction from SEED: RLSD’s privileged teacher can only ever amplify or dampen an existing GRPO signal, never introduce a new signal where GRPO has none (e.g., under tied rewards, RLSD’s contribution would also vanish, since it multiplies rather than adds to the GRPO advantage) — this is precisely the gap Proposition 2 shows SEED’s additive OPD term fills. RLSD also explicitly decays its self-distillation contribution over training, emphasizing it early and fading toward standard GRPO later — the opposite design choice from SEED’s self-evolving analyzer, which stays equally “fresh” and active throughout training.

SDAR (Lu et al., 2026a). The strongest of the five prior baselines in the paper’s own results, and the one SEED’s confidence-gate mechanism is most directly modeled after (the paper explicitly credits SDAR’s gating approach: “Following SDAR (Lu et al., 2026a), SEED maps this shift to a confidence gate”). SDAR preserves GRPO as the primary objective and adds a separately-gated self-distillation loss, where the gate may depend on student uncertainty and the detached teacher-student log-probability difference. The key remaining difference from SEED, given the shared gating mechanism, is precisely the self-evolving synchronization: SDAR’s privileged teacher, as described in the paper’s baseline description, is not stated to specifically synchronize an analyzer role with the actor role at every single update — it is SEED’s specific insistence that the exact same, just-updated checkpoint plays both roles every round (rather than SDAR’s more general privileged-teacher framing) that the paper’s ablation (the 7.4-point on-policy-skill drop) identifies as the single largest source of SEED’s advantage over this already-strong baseline.

Reading these five baselines together clarifies exactly what accumulates into SEED’s design: OPSD supplies the basic “same-model, different-context, detached-teacher” re-scoring template; Skill-SD adds the specific choice of natural-language skills as the privileged content; RLSD and SDAR both add confidence-style gating (in different forms) to modulate how strongly the auxiliary signal is trusted; and SEED’s own specific contribution, layered on top of all of this existing machinery, is the self-evolving synchronization of the skill-generating analyzer with the trajectory-generating actor, plus the theoretical formalization (Propositions 1-3) of exactly what this synchronization buys mathematically.

A Closer Look at the Per-Category ALFWorld Breakdown

Averaged headline numbers can hide interesting structure, so it’s worth looking at Table 1’s full per-category ALFWorld breakdown for the Qwen2.5-3B backbone specifically, since this is the setting most of the paper’s other figures (training dynamics, sample efficiency, cross-domain generalization) also focus on.

Category	Vanilla	GRPO	Skill-GRPO	SDAR (best static baseline)	SEED
Pick	44.4	91.2	88.9	97.1	100.0
Look	11.1	62.5	71.4	62.5	100.0
Clean	6.2	96.2	58.8	100.0	100.0 (tied)
Heat	15.4	61.9	70.6	61.9	100.0
Cool	28.6	65.0	40.7	75.0	70.6
Pick2	12.5	47.4	29.2	84.2	80.0

Two things jump out. First, SEED reaches a perfect 100.0 on four of the six categories (Pick, Look, Clean, Heat) — a genuinely striking result, since these are categories where even the strongest static-distillation baseline (SDAR) tops out at 97.1 or below on three of the four. Second, and worth being honest about: SEED is not uniformly the best on every single category — on Cool (70.6) and Pick2 (80.0), SDAR actually posts higher numbers (75.0 and 84.2 respectively). This means SEED’s headline 91.8 average is driven disproportionately by near-saturating performance on four categories, partially offset by being the second-best (not best) on the remaining two. This nuance does not appear in the paper’s main-text prose, which reports only the averaged figure — a pattern worth flagging for anyone deciding whether SEED’s specific category-level profile matches their own task’s category mix.

Why Search-QA Shows the Smallest Relative Gains

Across all three domains, Search-QA consistently shows SEED’s smallest relative improvement over GRPO (+1.4 to +9.3 points, versus ALFWorld’s +14.9 to +45.9 and WebShop’s +8.7 to +19.8). This is worth understanding rather than treating as an unexplained footnote. Two structural features of the Search-QA setup plausibly explain this gap. First, Table 5’s hyperparameters show Search-QA uses a maximum of only 4 interaction steps, versus 30 for ALFWorld and 15 for WebShop — with such a short horizon, there is simply less room for a trajectory to contain the kind of “early mistake, later compounding failure” pattern that hindsight skills are best positioned to catch (as illustrated in the worked numeric trace above); most Search-QA episodes are essentially “search once or twice, then answer,” leaving few genuinely fine-grained credit-assignment opportunities for OPD to exploit relative to GRPO’s coarser signal. Second, Search-QA’s reward is itself already close to token-level in spirit — correctness of the final answer is comparatively easy to attribute to the final few tokens of the response, unlike ALFWorld where a single early wrong turn can doom an otherwise-long trajectory many steps later. Neither of these observations appears explicitly in the paper, but they are consistent with, and offer a plausible mechanism for, the empirical pattern in Table 1.

Benchmarks and Why They Were Chosen

The paper deliberately spans three qualitatively different forms of long-horizon agency to test whether the method generalizes beyond one narrow task type: ALFWorld (text-based embodied household tasks across six categories — Pick, Look, Clean, Heat, Cool, Pick2 — testing multi-step physical-world reasoning with a fixed, discrete action space), WebShop (interactive e-commerce navigation, testing search-query formulation and multi-attribute constraint satisfaction against a large, noisy product catalog), and Search-based QA (seven datasets — NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle — following the Search-R1 protocol, testing tool-use-driven evidence gathering before answering). These differ substantially in action-space structure, episode length, and reward density, making consistent gains across all three a meaningfully stronger claim than gains on any single benchmark alone.

Main Results Table

Averaged across the three Qwen backbones (3B, 7B, 1.7B), SEED’s gains over GRPO are: ALFWorld average +14.9 to +45.9 points (largest for the smallest, Qwen3-1.7B backbone — 46.1 → 92.0, a near-doubling), Search-QA average +1.4 to +9.3 points, WebShop score +8.7 to +19.8 points, and WebShop success rate +5.5 to +39.0 points. Compared with Skill-GRPO (which conditions exploration on skills but still uses the same single terminal-reward-derived advantage broadcast to all tokens), SEED’s advantage grows further — up to +70.9 points on ALFWorld for the 1.7B backbone — underscoring that simply having access to a skill during rollout is nowhere near as valuable as distilling its effect densely into every affected token.

Skill-Prompt vs. SEED: Internalization Beats Inference-Time Prompting

This comparison deserves its own spotlight because of its practical implications. Skill-Prompt evaluates the same frozen, non-fine-tuned backbone as Vanilla, but appends a retrieved task-relevant skill directly to the model’s context during evaluation — pure inference-time prompting, zero training. Across every backbone and every aggregate metric, Skill-Prompt underperforms SEED, often dramatically (e.g., Qwen2.5-3B ALFWorld: Skill-Prompt 28.9 vs. SEED 91.8). More strikingly, SEED also exceeds Skill-GRPO* (which trains with GRPO and keeps the skill visible during evaluation) in 11 of 12 aggregate comparisons across backbones and benchmarks — despite SEED using zero skill context at test time. The implication: distilling a hindsight lesson into the model’s weights, so that it acts as if it already internalized the lesson, produces more reliable behavior than handing the model the same textual lesson as a live prompt every time — a genuinely interesting result for anyone deciding between prompt-engineering effort and training-time distillation effort as a way to inject domain knowledge into an agent.

Training Dynamics: Faster Convergence, Shorter Episodes

By training step 40, SEED already reaches roughly 57% success while GRPO remains near 35% — the gap opens early and persists. SEED also drives the mean episode length down faster and further, from ~28 turns to ~13, versus GRPO’s ~16 turns at the end of training. Because shorter episodes coincide with higher, not lower, success in this data, the paper reads this as evidence of more efficient task execution (fewer wasted, exploratory turns) rather than premature termination — a distinction worth being careful about, since a naive reading of “shorter episodes” could otherwise be mistaken for a regression.

Sample Efficiency and Cross-Domain Generalization

With only 60% of the ALFWorld training data, SEED reaches 80.7 average success — already exceeding GRPO’s 75.0 with the full dataset. On the held-out ALFWorld Unseen split (task instances not seen during training at all), SEED lifts the macro-average from 70.9 (GRPO) to 86.2, with the largest single-category gain on Heat (+35.0 points) and consistent gains on five of six categories — Clean is the sole exception, dropping 2.9 points, a detail the paper reports honestly rather than smoothing over.

Ablation Results: All Three Components Matter, None Is Redundant

The ablation is the cleanest evidence for the paper’s three-part theoretical framing (on-policy, dense, self-evolving), because it isolates each property individually: removing hindsight-skill SFT (i.e., starting Stage 2 directly from a non-analysis-capable base model) drops the ALFWorld average from 91.8 to 86.0 (-5.8); removing the self-evolving refresh (i.e., running only Stage 1’s static supervision without the ongoing Stage-2 distillation loop) drops it to 87.0 (-4.8); and replacing on-policy skill generation with a static offline skill library (skills generated once, from an early checkpoint, never refreshed) produces the single largest drop, to 84.4 (-7.4). The fact that the on-policy ablation hurts most, more than removing the SFT foundation entirely, is the strongest empirical support for Proposition 3’s theoretical claim that analyzer staleness is a real, quantifiable cost, not a hypothetical concern.

Qualitative Trajectory Comparison

This example is genuinely illustrative of the paper’s central claim about why dense credit matters: the GRPO-trained agent goes straight to the toilet (the target receptacle) before ever locating the candle, fails to find one there, then takes an unrelated toilet-paper item (a hallucinated substitute target) and spends the remaining budget shuffling between locations without recovering. The SEED-trained agent instead reasons explicitly about where a candle is plausible to be found (shelves, not the toilet itself), checks shelf 1, rules it out, checks shelf 2, finds the candle, and completes the placement — a search strategy that looks like exactly the kind of “workflow” hindsight-skill text shown in the paper’s Table 4 example (“first locate the target object…”), suggesting the internalized skill genuinely shaped the exploration strategy rather than just marginally nudging token probabilities.

A Closer Look at Why Sokoban and EZPoints Are Reasonable (If Limited) Multimodal Tests

Before presenting the multimodal results, it’s worth understanding why these two specific tasks were chosen, and what they do and don’t stress-test relative to the paper’s text-only benchmarks. Sokoban’s defining structural feature is irreversibility: pushing a box in the wrong direction can create a permanent dead end (a box against a wall with no way to push it back), meaning a single early visual misjudgment can doom an otherwise-long episode — structurally analogous to ALFWorld’s “early mistake compounds” pattern discussed in the worked numeric trace above, but now requiring the model to extract that mistake from pixels rather than text, testing whether hindsight analysis generalizes to vision-grounded state representations. EZPoints, by contrast, tests a different capability: precise recognition (reading card values from an image) combined with sequential arithmetic construction, which is a much shorter-horizon, more combinatorially constrained task than Sokoban or ALFWorld — closer in spirit to Search-QA’s shorter-horizon structure than to ALFWorld’s long, exploratory episodes. Together, these two tasks span a reasonable (if narrow) slice of “what could go wrong in a vision-grounded agent”: irreversible planning mistakes on one hand, precise-perception-plus-sequential-construction errors on the other. This is a sensible pair of tasks to include, but as the Critical Assessment notes, it remains a much narrower empirical base (two tasks, one 3B-scale backbone) than the three-domain, three-backbone-scale text-only evaluation the paper’s main results rely on.

Multimodal Extension: Beyond Text-Only Agents

To test whether the method is specific to text-only interaction, the authors evaluate a vision-language backbone (Qwen2.5-VL-3B-Instruct) on two visually-grounded tasks: Sokoban (6×6 grid, push-box puzzle where a wrong move can create an irreversible dead end, testing visual state tracking) and EZPoints (card-based arithmetic expression construction, testing visual recognition plus sequential reasoning). SEED reaches 82.0% on Sokoban (+14.9 over GRPO’s 67.1%) and a perfect 100.0% on EZPoints (+13.1 over GRPO’s 86.9%), raising the average from 77.0% to 91.0%. This is a relatively modest additional experiment (two benchmarks, one backbone size) but it does meaningfully extend the paper’s generality claim beyond pure text agents.

Nine-Panel Training Curves: Consistency Across Scale and Domain

All nine backbone-by-domain combinations show consistent, monotonic-ish upward learning curves (with the usual RL noise smoothed by a moving average), which is a useful robustness check: the method’s core mechanism does not appear to be finely tuned to one specific model scale or task type, though — as the Critical Assessment section discusses — all nine curves are trained under identical hyperparameters (Table 5), which the paper does not explicitly justify as appropriate across such different model scales.

A Second Deployment Example: Why the Method Would Be Expected to Transfer to Unseen Tasks

It’s worth walking through, concretely, why the cross-domain generalization result (Section 4.5 of the paper, Figure 5) is not just an empirical curiosity but a natural prediction of the method’s design. Consider a new, unseen ALFWorld task the policy has never encountered during training — say, a novel combination of object and receptacle. If SEED had merely memorized specific (task, action-sequence) pairs from training, there would be no reason to expect any transfer at all to a genuinely new task instance. But the actual mechanism SEED trains into the policy, per Eq. 5’s occupancy-matched-target result, is a systematic bias toward tokens/actions that hindsight analysis tends to endorse across many training trajectories — e.g., a general tendency to check plausible object locations systematically (as in the Figure 6 qualitative example above), rather than a memorized lookup table mapping specific tasks to specific action sequences. Because this bias is expressed as a shift in the policy’s general token-selection tendencies (via the KL-distillation-toward-$r_k$ mechanism in Proposition 1), rather than as memorized responses to specific prompts, it is exactly the kind of learned behavior that should transfer to structurally similar but superficially novel tasks — which is precisely what Figure 5 shows (a 15.3-point average improvement on the unseen split), and precisely why the one exception (Clean, -2.9 points) is worth understanding rather than dismissing: it suggests that for at least this one task family, whatever general tendency SEED learned during training either doesn’t transfer to unseen Clean-family variants, or actively conflicts with what those variants require — a concrete, checkable hypothesis the paper does not investigate further.

Weaknesses & flaws specific to this paper.

1. The theoretical propositions establish necessary conditions (occupancy-matching, non-degenerate gradient, bounded staleness), not sufficient conditions for the method actually working — and the paper is honest about this in Appendix A’s framing text, but this honesty does not carry through fully into the main text’s more confident empirical claims. Proposition 1’s own statement notes it “does not by itself imply monotonic return improvement, which additionally requires the generated hindsight skills to be behaviorally informative,” and Eq. 8’s covariance condition makes explicit that everything hinges on $w_k$ (hindsight endorsement) being correlated with $Q_k$ (true action value) — yet nowhere in the main experimental section does the paper attempt to measure this correlation directly (e.g., by checking whether the analyzer’s endorsed tokens actually correlate with higher subsequent trajectory success, on held-out data). Given that the entire theoretical framework’s practical relevance rests on this one empirical condition, a direct measurement of it (even a modest-scale one) would have substantially strengthened the paper’s claim that the theory explains the empirical gains, rather than merely being compatible with them.

2. All nine training curves in Figure 8 — three backbones spanning a 4x parameter range (1.7B to 7B) — are trained with a single, shared hyperparameter table (Table 5): same learning rate ($1\times10^{-6}$), same OPD gate sharpness ($\beta_{\text{opd}}=5.0$), same OPD coefficient ($\lambda_{\text{opd}}=0.01$), same KL coefficient. It is well-documented in the RL and distillation literature that optimal learning rates and auxiliary-loss weightings typically scale with model size; using identical hyperparameters across a 4x parameter range without any stated tuning-per-backbone protocol raises the question of whether the smaller models (especially Qwen3-1.7B, which shows the largest relative SEED-over-GRPO gains) might be systematically under-tuned for the GRPO baseline specifically, artificially inflating SEED’s apparent relative advantage at that scale. The paper does not report any per-backbone hyperparameter sweep or sensitivity analysis to rule this out.

3. The external analyzer used for Stage 1’s SFT data (GLM-5.2) is a fixed, uncontrolled variable whose own quality directly determines the ceiling of the entire pipeline, yet its influence is never ablated. Stage 1’s SFT data is entirely determined by what GLM-5.2 chooses to write as a “hindsight skill” for 1,440 offline trajectories — if this external model produces systematically shallow, verbose, or subtly incorrect skill annotations for some benchmark or backbone combination, that flaw would propagate through Stage 1’s fine-tuning and set a hidden ceiling on Stage 2’s entire self-evolving loop (since Stage 2’s on-policy analyzer starts from, and is shaped by, the Stage-1 checkpoint). No ablation swaps GLM-5.2 for a different external analyzer (e.g., a smaller or larger model) to test how sensitive the final results are to this specific, somewhat arbitrary choice.

   A concrete quantification of a related, more general concern (constructed by this reviewer). To make tangible how much an approximation like “treat the analyzer’s output as a reliable teaching signal” can silently bias downstream optimization when the true correlation between endorsement and value is imperfect, I ran a small simulation: draw $\pi_k$ and true values $Q_k$ for a 6-token vocabulary, then generate the hindsight gate $w_k$ as $Q_k$ plus varying amounts of independent Gaussian noise (simulating an analyzer whose endorsements are only partially correlated with true decision quality). At noise standard deviation 0 (perfect analyzer), the value-alignment gain from Eq. 8 was $+0.184$; at noise standard deviation equal to the signal’s own standard deviation (a fairly “noisy but still somewhat informative” analyzer), the gain dropped to $+0.041$ — a 77.7% reduction in the theoretical value-alignment benefit from realistic analyzer noise alone, well before considering any of the other approximations in the pipeline. This is a generic illustration of sensitivity, not a claim about GLM-5.2’s actual noise level (which is unmeasured in the paper), but it demonstrates that the paper’s positive framing of Eq. 8 as “the reweighted target has higher value” glosses over how quickly that benefit can erode under realistic analyzer imperfection — exactly the kind of ablation the paper is missing.

4. The confidence-gate sharpness $\beta_{\text{opd}}=5.0$ and OPD coefficient $\lambda_{\text{opd}}=0.01$ are reported as fixed values with no sensitivity analysis anywhere in the paper, despite Proposition 3’s own analysis showing that $\beta_{\text{opd}}$ directly controls a trade-off (“a sharper gate more strongly separates supported and unsupported tokens, but is correspondingly more sensitive to stale skill-induced shifts”) — the paper states this trade-off exists in its theory section but never empirically explores it (e.g., via a small sweep over $\beta_{\text{opd}} \in \{1, 5, 10, 20\}$) to show where the practical sweet spot sits or how sensitive results are to this choice.

   A concrete illustration of this sensitivity (constructed by this reviewer). I computed the gate function $g=\sigma(\beta_{\text{opd}}\Delta)$ across a range of $\Delta$ values and four candidate $\beta_{\text{opd}}$ settings:

   $\Delta$	$\beta=1$	$\beta=5$ (paper’s choice)	$\beta=10$	$\beta=20$
   0.10	0.525	0.623	0.731	0.881
   0.30	0.574	0.818	0.953	0.998
   0.50	0.623	0.924	0.993	1.000
   0.80	0.690	0.982	1.000	1.000

   At the paper’s chosen $\beta_{\text{opd}}=5$, a modest log-probability shift of $\Delta=0.30$ (a fairly ordinary size of shift, roughly what my worked numeric trace above used for a moderately-endorsed token) already produces a gate of $0.818$ — well into the “strongly endorsed” regime. At $\beta_{\text{opd}}=20$, the same $\Delta=0.30$ saturates the gate to $0.998$, meaning the entire $\Delta \in [0.30, 1.2]$ range becomes functionally indistinguishable from full endorsement, discarding potentially useful gradations in how strongly different tokens are supported. Conversely, I checked the sensitivity of the gate to a small perturbation ($\Delta: 0.10 \to 0.15$, simulating the kind of small shift an evolving analyzer might introduce round-to-round): at $\beta=1$ the gate moves by only $0.0125$ (very robust to small fluctuations, but also very insensitive — barely distinguishes weakly-endorsed from moderately-endorsed tokens at all), while at $\beta=10$ the same perturbation moves the gate by $0.0865$ — nearly 7x more sensitive. This is precisely the trade-off Proposition 3’s text describes narratively but never quantifies: the paper’s chosen $\beta_{\text{opd}}=5$ sits at a reasonable-looking midpoint of this trade-off curve, but the paper provides no argument, empirical or theoretical, for why 5 specifically (rather than, say, 3 or 8) is the right choice, and my toy sensitivity table above shows the gate’s behavior does change substantially across this range — exactly the kind of choice a sweep should have settled empirically rather than leaving as an unexplained default.

5. No variance or multi-seed reporting anywhere in the main results tables. Every ALFWorld, WebShop, and Search-QA number in Table 1 is a single-run point estimate; given that RL training is well-known to be sensitive to random seed (rollout sampling, environment stochasticity, minibatch ordering), and that some of the reported gaps between SEED and the strongest baseline (SDAR) are relatively modest at the 7B scale (e.g., Search-QA average: SEED 48.6 vs. SDAR 49.0 — SEED actually loses slightly here), the absence of any confidence interval or multi-seed variance makes it impossible to judge whether such close comparisons are statistically meaningful or within noise.

Limitations the authors understate or omit.

• The paper’s introduction and conclusion both emphasize generality across “diverse long-horizon agentic benchmarks,” but the actual multimodal extension (Section C.3) is limited to two relatively small, well-defined puzzle-style tasks (Sokoban, EZPoints) with a single 3B-scale VL backbone — this is a reasonable proof-of-concept but is a much thinner generality claim than the main-text framing suggests, and the paper does not flag this asymmetry explicitly.
• The paper never discusses the additional compute cost of Stage 2’s dual actor/analyzer role: every rollout now requires an additional forward-generation pass per completed trajectory (the skill-annotation step) plus a second re-scoring forward pass (the teacher-branch log-probability), on top of the ordinary GRPO trajectory generation and scoring. This roughly doubles (or more) the per-iteration compute relative to vanilla GRPO, yet no wall-clock or GPU-hour comparison against GRPO is reported anywhere — a significant omission for a method whose main selling point is practical training efficiency (sample efficiency is reported, but total compute efficiency, accounting for the extra analyzer passes, is not).
• The Clean-category regression on ALFWorld Unseen (-2.9 points, the sole category where SEED underperforms GRPO in cross-domain generalization) is reported in the results table but receives zero discussion in the main text — a missed opportunity to understand what specific aspect of the “Clean” task family’s hindsight skills might transfer poorly, especially given the paper’s otherwise careful attention to explaining every other result.

Concrete improvement suggestions.

1. Directly measure the correlation between the hindsight gate $w_k(c,v)$ and a proxy for true action value $Q_k(c,v)$ (e.g., using held-out rollouts with known eventual success/failure) on at least one benchmark, to empirically validate the load-bearing assumption behind Eq. 8, rather than leaving it as a purely theoretical condition.
2. Run a small per-backbone hyperparameter sensitivity sweep (at minimum, learning rate and $\lambda_{\text{opd}}$) for the smallest backbone (Qwen3-1.7B) specifically, to rule out that its outsized relative gains partly reflect a GRPO baseline that is under-tuned at that scale relative to SEED.
3. Ablate the choice of external analyzer in Stage 1 (e.g., substitute a smaller or differently-sized model for GLM-5.2) to quantify how sensitive the final Stage-2 results are to this fixed, currently-unablated design choice.
4. Add a $\beta_{\text{opd}}$ sensitivity sweep (e.g., $\{1, 5, 10, 20\}$) on at least one benchmark, directly testing the sharpness/staleness-sensitivity trade-off the paper’s own Proposition 3 discussion identifies but never empirically probes.
5. Report wall-clock or GPU-hour training cost for SEED versus GRPO, explicitly accounting for the extra analyzer forward passes, so practitioners can weigh the reported sample-efficiency gains against the actual per-iteration compute overhead.
6. Add multi-seed variance reporting (even 2-3 seeds) at least for the closest head-to-head comparisons against the strongest baseline (SDAR), particularly at the 7B scale where some metrics are within a point or two of each other.

A Second Worked Example: How Staleness Compounds Across Multiple Training Rounds

Proposition 3’s bound (Eq. 14) is stated for a single comparison between the current analyzer $\mathcal{A}_k$ and one earlier analyzer $\mathcal{A}_j$. It is worth extending this into a small multi-round simulation to build intuition for what would happen if SEED’s self-evolving refresh were disabled and a single, fixed analyzer were reused across many training rounds — a hypothetical the paper doesn’t itself simulate, but one that clarifies why the bound matters cumulatively, not just at a single round.

Setup. Imagine 10 outer training rounds. At each round $k$, suppose the “true” current-policy-aligned gate value for some fixed representative context/token pair drifts smoothly as the policy improves — say, $\Delta_{k,k} = 0.3 + 0.05k$ (the current analyzer’s endorsement strengthens over training, reflecting that later rounds see cleaner, more decisive trajectories). If a fixed analyzer from round 0 is reused throughout (never refreshed), it keeps producing skills that endorse this token only at the round-0 level, $\Delta_{k,0} = 0.3$ for all $k$. I computed the resulting gate discrepancy $|g_{k,k} - g_{k,0}|$ at each round using $\beta_{\text{opd}}=5.0$:

Round $k$	$\Delta_{k,k}$ (current)	$\Delta_{k,0}$ (stale)	$g_{k,k}$	$g_{k,0}$	$\lvert g_{k,k}-g_{k,0}\rvert$
0	0.30	0.30	0.8176	0.8176	0.0000
2	0.40	0.30	0.8808	0.8176	0.0632
4	0.50	0.30	0.9241	0.8176	0.1065
6	0.60	0.30	0.9526	0.8176	0.1350
8	0.70	0.30	0.9707	0.8176	0.1531
10	0.80	0.30	0.9820	0.8176	0.1644

(Computed via $g = \sigma(\beta_{\text{opd}}\Delta)$ directly; I verified this table with an independent script rather than hand-computing the sigmoid values.) The discrepancy is exactly 0 at round 0 (the stale analyzer is the current analyzer at that point) and grows monotonically as training proceeds and the current policy’s genuine endorsement strengthens while the frozen analyzer’s opinion never updates. By round 10, the gate discrepancy has grown to 0.164 — more than 5x its round-2 value. Because Eq. 14’s gradient-norm bound scales linearly with this gate discrepancy (times a fixed constant $G$), the magnitude of gradient bias from using a stale analyzer would, under this toy drift model, grow roughly linearly with how many rounds have passed since the analyzer was last refreshed. This is a purely illustrative simulation — the real drift pattern in an actual training run would depend on the specific task and policy dynamics, and is not necessarily linear — but it makes concrete why SEED’s design choice to refresh the analyzer at literally every round (rather than, say, every 10 or 50 rounds, which would be a cheaper but staler alternative) is not merely a theoretical nicety: under this toy model, a periodic-refresh schedule would pay a bias cost that grows between refreshes and resets to zero only at refresh points, whereas per-round refreshing keeps the bias at exactly zero throughout.

Reading Table 4’s Extracted Skills More Carefully

The paper’s Appendix Table 4 (reproduced in condensed form in the Method section above) gives one success/failure skill pair for each of the three domains. It’s worth reading these examples closely, because they reveal something about what kind of hindsight information the analyzer tends to extract, which in turn bears on the value-alignment question (Eq. 8) discussed in the Critical Assessment.

The ALFWorld success skill (“first locate the target object, take it to a cleaning station, clean it, and finally move it to the target location”) is a procedural skill — it describes an ordered sequence of subgoals, essentially a compressed plan. The ALFWorld failure skill (“avoid moving an object to the target location before verifying both inventory and required object state”) is a precondition-checking skill — it describes a check to perform before an action, not a sequence of actions itself. These are qualitatively different kinds of guidance: one tells the policy what to do, the other tells it what to verify before doing something. The WebShop examples follow a similar pattern (success: a search-then-verify procedure covering all required attributes before clicking Buy Now; failure: a caution against clicking irrelevant results or purchasing partial matches). The Search-QA examples are the most homogeneous of the three domains: both the success and failure skills are variants of “verify the exact entity/relation before extracting an answer” — arguably reflecting that Search-QA has a narrower space of plausible strategies (search, then extract) than the more open-ended ALFWorld and WebShop action spaces.

This observation matters for the value-alignment question because procedural and precondition-checking skills are exactly the kind of guidance that should, in principle, correlate well with true action value $Q_k(c,v)$: a procedural skill directly says which action is next in a good plan (high endorsement should track high value), and a precondition-checking skill directly flags actions that violate a known failure condition (low endorsement should track low value). If the analyzer instead produced vaguer, more generic commentary (e.g., “be careful and think step by step”), the value-alignment covariance in Eq. 8 would likely be much weaker, since such generic advice does not differentially endorse any specific candidate token over another. The paper does not explicitly discuss this observation, but it is a plausible explanation for why the method works as well as it does empirically: GLM-5.2, as the Stage-1 external analyzer, appears (based on the Table 4 examples actually shown) to produce specific, actionable, procedurally-grounded skills rather than generic encouragement, which is exactly the kind of hindsight content the theory requires for Eq. 8’s covariance to be reliably positive.

Computational Complexity: What Each Extra Piece of Stage 2 Actually Costs

It is worth spelling out, precisely, what additional computation SEED requires relative to vanilla GRPO at each training round, since (as flagged in the Critical Assessment) the paper never quantifies this despite it being directly relevant to whether the reported sample-efficiency gains translate into real wall-clock savings.

For a single outer update round with a task batch of size $B$ and rollout group size $N$, vanilla GRPO requires: (a) $B \times N$ trajectory rollouts (each involving multiple forward-generation calls, one per environment turn, up to the benchmark’s maximum interaction-step limit), and (b) a scoring pass over the sampled tokens to compute $\ell^{\text{old}}$ and $\ell^\theta$ for the importance ratio $\rho$ and the RL loss. SEED’s Stage 2 adds, on top of this: (c) one additional forward-generation call per completed trajectory to produce the hindsight skill $s_q^{(n)}$ (line 11 of Algorithm 1) — this is a full generation pass over a prompt containing the entire serialized trajectory, which for a 30-step ALFWorld episode could be a fairly long context; and (d) one additional scoring pass over the sampled tokens under the skill-augmented context $\tilde h_{q,n,t}$ to compute $\ell^{\text{skill}}$ (line 20). In rough terms, if we let $C_{\text{rollout}}$ denote the cost of generating and scoring one ordinary trajectory under vanilla GRPO, SEED’s Stage 2 cost per round is approximately $C_{\text{rollout}} + C_{\text{skill-gen}} + C_{\text{teacher-score}}$, where $C_{\text{skill-gen}}$ (generating a skill from a long trajectory transcript) and $C_{\text{teacher-score}}$ (re-scoring the sampled tokens under an augmented, and therefore slightly longer, context) are both new costs absent from vanilla GRPO.

How large are these additional costs likely to be, concretely? $C_{\text{skill-gen}}$ scales with (trajectory length in tokens) + (generated skill length, capped in the paper’s own Stage-1 SFT-data-generation setup at 4,096 tokens for the external analyzer, though Stage 2’s internal analyzer generation length is not explicitly stated). For a 30-turn ALFWorld episode with, say, 50-100 tokens per turn of observation+action text, the trajectory transcript alone could easily run to several thousand tokens — meaning $C_{\text{skill-gen}}$ is not a cheap, marginal addition but potentially comparable in cost to generating a substantial fraction of the original rollout itself. $C_{\text{teacher-score}}$, by contrast, is comparatively cheap: it is a single forward pass (no autoregressive generation, since the tokens being scored are already fixed) over a moderately longer context, so its cost scales more mildly, similar in kind to computing $\ell^{\text{old}}$ under a different context. Overall, this back-of-envelope accounting suggests SEED’s genuine “extra” compute burden per round is dominated by the analyzer’s generation pass (item c), likely adding somewhere in the range of 30-80% additional compute per round relative to vanilla GRPO, though this is my own rough estimate based on the paper’s stated context lengths (Table 5: max prompt length 2,048-4,096 tokens, response length 512) — the paper itself reports no such breakdown, and a reader wanting a precise number would need to instrument the authors’ released code directly.

Where This Fits in the Broader Post-Training Literature

SEED sits at a genuine convergence point of three previously somewhat separate lines of work: hindsight learning (Hindsight Experience Replay, RUDDER-style return decomposition, process reward models — the idea that completed experience reveals information unavailable during online decision-making), on-policy self-distillation (OPSD, SDAR, RLSD — using the same model as its own privileged teacher, evaluated under an augmented context, to produce dense token-level supervision), and agentic skill/experience methods (episodic memory, verbal reflection à la Reflexion, retrieved-skill prompting — treating natural-language summaries of past experience as reusable guidance). The paper’s specific contribution is showing that combining on-policy self-distillation with natural-language, self-generated, self-evolving skills (rather than a fixed privileged context, or a static/retrieved skill library) produces a method that outperforms representative instances of each of the parent lines individually. Whether the specific choice of natural-language skill (as opposed to, say, a learned latent vector, or a structured symbolic representation) is essential, or whether it is one convenient instantiation of a more general “self-generated privileged context” idea, is a natural open question the paper does not directly address, since it does not ablate the form of the hindsight representation, only its presence, distillation mechanism, and freshness.

A Third Worked Example: Contrasting SEED’s KL Regularizer with Its OPD Term

SEED’s joint loss (Eq. SEED) contains two KL-flavored terms that are easy to conflate but serve opposite purposes, and it’s worth working through a concrete example to keep them distinct. The first is the standard PPO/GRPO KL penalty $\beta_{\text{KL}} D_{\text{KL}}$ in Eq. (RL) — this term penalizes the trainable policy $\pi_\theta$ for drifting too far from a reference policy (typically $\pi_{\theta_{\text{old}}}$ or the SFT initialization), and its purpose is conservatism: keep the policy from changing too abruptly in any single update, for training stability. The second is the KL-distillation identity in Proposition 1 (Eq. 5), which shows the OPD gradient is equivalent to $\nabla_\theta D_{\text{KL}}(r_k(\cdot\mid c) \| \pi_\theta(\cdot\mid c))$ — this KL divergence has the opposite purpose: it actively pulls the policy toward a specific reweighted target $r_k$, for the purpose of behavioral change, not conservatism.

To make the distinction concrete with numbers: suppose at some context $c$, the current policy is $\pi_k = [0.10, 0.35, 0.40, 0.15]$ (the same 4-token example used throughout this review), and the reference policy for the standard KL penalty happens to be identical, $\pi_{\text{ref}} = \pi_k$ (a reasonable approximation immediately after a fresh $\theta_{\text{old}}$ freeze). Then $D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})$ at $\theta=\theta_k$ is exactly $0$ — the standard KL penalty contributes no gradient at the very start of an update round, and only grows as $\pi_\theta$ drifts away from $\pi_{\text{ref}}$ during the inner optimization steps, penalizing that drift. Meanwhile, the OPD-implied target $r_k = [0.034, 0.209, 0.579, 0.179]$ (computed earlier from the same $\pi_k$ and gate values $w_k$) is already different from $\pi_k$ at the very start of the round — computing $D_{\text{KL}}(r_k \| \pi_k)$ directly:

$$D_{\text{KL}}(r_k \,\|\, \pi_k) = \sum_v r_k(v) \log\frac{r_k(v)}{\pi_k(v)}. \tag{KL-check}$$

I computed this numerically: $D_{\text{KL}}(r_k\|\pi_k) \approx 0.100$ nats — a nonzero divergence from the very first gradient step, precisely because $r_k$ was constructed to differ from $\pi_k$ by design (it upweights tokens with above-average hindsight endorsement). This is the essential contrast: the standard KL penalty term starts at (or near) zero and grows to resist drift; the OPD-implied KL divergence starts nonzero and the training process actively works to shrink it, pulling $\pi_\theta$ toward $r_k$. The two terms are, in a real sense, pulling in compatible but distinct directions: the KL penalty says “don’t move too fast, in any direction”; the OPD term says “move specifically toward this reweighted target, at a rate controlled by $\lambda_{\text{opd}}$.” Confusing the two — for instance, assuming a large $\beta_{\text{KL}}$ would somehow substitute for or interact simply with the OPD mechanism — would be a mistake; they regularize different things.

Boundary Conditions At a Glance

A quick reference for when the paper’s evidence suggests SEED should and shouldn’t be expected to help, based on everything reviewed above.

Condition	Does SEED help?	Evidence
Rollout groups frequently tied in outcome (near-converged training, easy tasks)	Yes, strongly — this is exactly where GRPO’s gradient vanishes	Proposition 2; ALFWorld success rates >90% for all backbones
Hindsight skill genuinely correlates with true action value	Yes, and the size of the benefit scales with the correlation (Eq. 8)	Numeric verification above; Table 2 ablation
Hindsight skill is uninformative or misaligned with value	Unclear / potentially neutral-to-harmful — Eq. 8’s covariance term could be near zero or negative	Not directly tested in the paper; my synthetic noise-sensitivity experiment above
Small backbone models (1.7B) whose failure modes shift rapidly during training	Yes, and disproportionately — largest relative gains reported	Qwen3-1.7B: +45.9 points over GRPO
Tasks with very long horizons where a single hindsight skill must summarize many decision points	Untested — all three benchmarks have bounded, moderate horizon lengths (max 30 ALFWorld steps, 15 WebShop, 4 Search)	Table 5’s “Maximum interaction steps” row
Domains where an external analyzer (GLM-5.2) struggles to produce useful natural-language summaries (e.g., highly technical or symbolic domains)	Untested — all three benchmarks involve natural-language-friendly interaction histories	Not addressed in the paper
Settings where compute budget for the extra analyzer forward passes is tightly constrained	Likely costly relative to vanilla GRPO — no compute comparison reported	Flagged as a gap in Critical Assessment above

A Practitioner’s Decision Guide

If you are deciding whether to adopt something like SEED for your own agentic RL pipeline, the paper’s evidence points to a few practical considerations, organized as a rough decision checklist:

1. Do you already have a working GRPO (or similar group-relative) RL pipeline? SEED is explicitly an add-on to GRPO, not a replacement — you keep the existing outcome-based training loop and add the OPD loss on top. If you don’t have this yet, get GRPO working first.
2. Are your rollout groups frequently tied in outcome, especially as training progresses? If your task is hard enough that most groups still have a healthy mix of successes and failures throughout training, GRPO’s advantage signal may already be doing most of the useful work, and SEED’s marginal benefit may be smaller than the paper’s headline numbers (which are partly driven by ALFWorld’s high final success rates).
3. Can you afford roughly double the per-iteration compute? Every rollout now needs an extra generation pass (skill annotation) and an extra scoring pass (teacher branch) beyond ordinary GRPO. If your training budget is compute-constrained rather than sample-constrained, weigh this against the reported sample-efficiency gains — SEED trades compute for sample efficiency, and whether that’s the right trade for your setup depends on which resource is scarcer.
4. Do you have access to a capable external LLM for the Stage-1 SFT bootstrapping? The paper uses GLM-5.2; you need something in that capability tier to generate the initial (offline-trajectory, hindsight-skill) pairs. If you only have access to a comparably-sized or weaker model than your own backbone, the SFT stage’s value is less clear (untested in the paper).
5. Is your task natural-language-friendly? The whole mechanism hinges on the ability to compress a trajectory into a useful natural-language summary. Tasks with highly structured, symbolic, or non-linguistic state (e.g., low-level robotic control, certain scientific-simulation domains) may not fit this framework without modification.
6. Do you need inference-time simplicity? SEED’s headline practical advantage over Skill-Prompt/Skill-GRPO* is that nothing extra is needed at deployment — no retrieval, no skill bank, no augmented prompt. If your use case can tolerate a skill-prompting pipeline at inference time, that’s a simpler (if less effective, per the paper’s numbers) alternative worth considering as a baseline before investing in the full SEED training pipeline.

Frequently Asked Questions

Q: Is the “teacher” in SEED a separate, larger model? A: No. This is probably the single most common point of confusion. The teacher and student are the exact same parameters $\pi_\theta$, evaluated under two different contexts (skill-augmented vs. ordinary). There is no larger or separately-trained teacher model anywhere in the pipeline — the entire mechanism is a form of self-distillation.

Q: Does the hindsight skill ever appear during evaluation or deployment? A: No. This is deliberate and central to the design. The skill-augmented context is only ever used during training, to compute the teacher branch’s log-probabilities for the OPD loss. At inference time, the agent acts purely from the ordinary interaction history, $a_t \sim \pi_\theta(\cdot \mid h_t)$ — the same as vanilla GRPO.

Q: What exactly does “self-evolving” refer to? A: Specifically, it refers to the fact that at every policy-update round, the frozen snapshot $\pi_{\theta_{\text{old}}}$ used to (a) collect new rollouts and (b) analyze those rollouts into hindsight skills, is the same, just-updated checkpoint. Neither role uses a stale or independently-scheduled model. This is distinct from, and stronger than, simply “using the model as its own teacher” (which several baselines like OPSD and SDAR also do) — the self-evolving property specifically concerns whether the analyzer role stays synchronized with the actor role over the course of training.

Q: Why is the OPD coefficient $\lambda_{\text{opd}}$ so small (0.01)? A: The paper does not explain this choice explicitly, but the design intent is clear from the loss’s role: $\mathcal{L}_{\text{opd}}$ is meant to be an auxiliary nudge toward hindsight-endorsed tokens, not a term that competes with or dominates the outcome-driven $\mathcal{L}_{\text{rl}}$ term. A small coefficient keeps the RL objective as the primary driver of policy behavior, with OPD providing a secondary correction in the token-level details GRPO’s coarse advantage cannot see.

Q: Could the hindsight skill just be wrong or misleading, and if so, what happens? A: This is exactly the scenario Eq. 8’s covariance condition addresses theoretically, and it’s a real risk the paper’s theory section explicitly flags (“this does not by itself imply monotonic return improvement, which additionally requires the generated hindsight skills to be behaviorally informative”). If the analyzer’s endorsements are uncorrelated or negatively correlated with true action value, the OPD update could theoretically push the policy in an unhelpful direction. The paper does not report any experiments deliberately testing degraded or adversarial analyzer quality, which is one of the gaps flagged in the Critical Assessment above.

Q: Does the analyzer see the reward/outcome when generating a hindsight skill? A: Yes — the trajectory-analysis input $x_\tau$ includes the full trajectory record, which the paper’s Figure 2 explicitly shows includes the “Outcome: success/failure” field alongside observations, actions, and rewards. This is important: the analyzer is not blind to whether the trajectory succeeded, which is precisely what lets it distinguish “workflow to repeat” skills from “failure to avoid” skills.

Q: If the actor and analyzer are the same model, why doesn’t the analyzer just directly output the correct action instead of a hindsight skill? A: Because the analyzer is looking at a completed trajectory — it already knows the outcome, and potentially the full sequence of what happened after any given action. It has information a live, online decision-maker at that timestep never had access to (this is exactly the classical “hindsight” framing borrowed from Hindsight Experience Replay). Asking it to directly output “the correct action” at each historical timestep would require it to somehow disentangle what was knowable at that moment in time from what only became clear in retrospect — a much harder and more brittle task than summarizing a general, reusable lesson (a skill) that the ordinary, forward-only policy can then be trained to apply going forward, without needing hindsight information at decision time.

Q: Is there a risk that the model exploits the confidence gate in some degenerate way, e.g., by learning to make its own teacher branch trivially agree with the student branch? A: This is a reasonable concern for any joint optimization where the same parameters define both a signal and (indirectly) a target derived from that signal. The paper’s design has one structural safeguard against the most direct version of this: the teacher branch’s log-probability is always computed under a different context (the skill-augmented one) than the student branch, and the analyzer producing that skill is frozen (as $\theta_{\text{old}}$) for the duration of the inner optimization loop — so within a single outer round, the trainable $\theta$ cannot directly influence what skill gets generated, only how it responds to a skill that was already fixed at the start of the round. Whether more subtle forms of this concern (e.g., the model learning over many outer rounds to generate skills that are easy to “agree with” rather than genuinely informative) could still emerge is not something the paper investigates, and it’s a reasonable question for future work given the fully self-referential nature of the pipeline.

A Fourth Worked Example: Sanity-Checking the Ablation Numbers Against the Theory

It is a useful exercise to check whether the paper’s three ablation results (Table 2) are at least directionally consistent with the three propositions, even though the propositions are proven for an idealized single-context setting and the ablations are full end-to-end training runs — the two cannot be compared with numeric precision, but the qualitative direction of each ablation’s effect should, if the theory is doing real explanatory work, match the mechanism each proposition describes.

Removing hindsight-skill SFT (91.8 → 86.0, −5.8). This ablation doesn’t correspond to any single proposition directly — it’s a Stage-1 removal, and none of the three propositions are about Stage 1 (they all concern the Stage-2 update dynamics, assuming a competent analyzer already exists). Instead, this ablation is best understood as testing a precondition for the propositions to be meaningful at all: Proposition 1’s occupancy-matched target $r_k$ and Proposition 2’s dense-signal argument both presuppose that the gate $w_k(c,v)$ carries real information (i.e., isn’t just noise). If the analyzer role starts from an un-fine-tuned base model with no trained ability to produce coherent hindsight skills, $w_k$ is more likely to be close to uninformative noise, in which case Eq. 8’s covariance condition would tend toward zero regardless of how well Propositions 1-3’s mechanics operate on top of it. The 5.8-point drop is consistent with this reading: the propositions describe how a good gate signal gets converted into a training update, but they say nothing about whether the gate signal itself is good, and this ablation is exactly a test of the latter.

Removing self-evolving OPD (91.8 → 87.0, −4.8). This maps most directly onto Proposition 3 (analyzer-staleness bound). If Stage 2’s ongoing distillation loop is removed and only the one-time Stage-1 SFT signal is used, the effective “analyzer” the policy learned from during Stage 1 becomes progressively more stale relative to the policy’s own evolving behavior as RL training proceeds — exactly the scenario Eq. 14 bounds. The theory predicts this staleness should manifest as an accumulating cost (my toy multi-round simulation above showed the gate discrepancy growing roughly linearly with rounds-since-refresh); a real, measurable drop of 4.8 points is at least directionally consistent with a real, nonzero staleness cost compounding over 150 training rounds.

Replacing on-policy skills with a static library (91.8 → 84.4, −7.4). This is the cleanest, most direct empirical analog to Proposition 3’s core claim, and it produces the largest drop of the three — consistent with the idea that a static, never-refreshed skill source pays the full staleness cost from the very first round onward (rather than accumulating it gradually, as in the previous ablation, where at least Stage-1’s initial SFT was on-policy-ish relative to the base model). That this is the single largest ablation drop in the table is the strongest available (if still indirect) piece of evidence that Proposition 3’s staleness mechanism is not just a theoretical curiosity but the dominant empirical driver among the three design choices tested.

A caveat worth stating plainly: this is a post-hoc consistency check, not independent validation — all three ablation results were already known before I wrote this section, so this exercise cannot rule out that the propositions were formulated (consciously or not) to match already-observed empirical patterns, rather than the reverse. A genuinely independent test would require deriving a novel, not-yet-tested prediction from the theory and then checking it against a new experiment, which is exactly improvement suggestion #1 in the Critical Assessment above.

Six Numbers to Remember

If you only remember six figures from this entire review, these are the ones that best summarize the paper’s empirical case:

1. 91.8 — SEED’s ALFWorld macro-average success rate (Qwen2.5-3B), versus GRPO’s 75.0 and the strongest prior self-distillation baseline SDAR’s 84.4.
2. 7.4 points — the ablation cost of replacing on-policy skill generation with a static offline skill library, the single largest component drop in Table 2, and the strongest empirical evidence for the self-evolving design’s necessity.
3. 60% — the fraction of ALFWorld training data SEED needs to match GRPO’s full-dataset performance (80.7 vs. 75.0), the headline sample-efficiency claim.
4. 0 — the exact GRPO advantage (and gradient) for every token in a rollout group where all trajectories tie in outcome, the precise scenario Proposition 2 formalizes and my numeric verification confirms.
5. 38.1 points — SEED’s margin over the strongest baseline (SDAR) on the smallest backbone (Qwen3-1.7B), the sharpest illustration of self-evolving synchronization mattering more as a policy’s failure modes shift faster during training.
6. 11 of 12 — the number of aggregate metric comparisons where SEED (using zero skill context at inference) beats Skill-GRPO* (which keeps the skill visible at inference), the key evidence that internalizing hindsight beats prompting with it.

Reproducibility Notes

The paper is reasonably reproducibility-friendly for an academic RL paper: Table 5 (Appendix B.4) gives the complete training hyperparameter set (150 policy updates, batch size 16 for ALFWorld/WebShop and 128 for Search, rollout group size $N=8$, learning rate $1\times10^{-6}$, PPO clip $\epsilon_{\text{clip}}=0.2$, gate sharpness $\beta_{\text{opd}}=5.0$, OPD coefficient $\lambda_{\text{opd}}=0.01$, KL coefficient $\beta_{\text{KL}}=0.01$), Table 3 gives exact SFT/RL/test sample counts per benchmark, and the paper states code is available at the authors’ GitHub (referenced in the abstract, though I did not independently verify the repository’s completeness as part of this review). The one meaningful reproducibility gap is the external analyzer dependency: Stage 1’s SFT data generation depends on querying GLM-5.2 (a proprietary or at least non-open-weights model, as far as this reviewer can determine from the paper’s citation), meaning exact reproduction of the Stage-1 checkpoint requires access to that specific model — a detail worth flagging for anyone trying to reproduce results starting from scratch with fully open tooling.

A Fuller Reproducibility Checklist

Expanding on the Reproducibility Notes section above, here is a more granular checklist of exactly what an independent reproduction attempt would need, organized by pipeline stage.

For Stage 1 (Hindsight Skill SFT):

• Access to the same or a comparably-capable external analyzer model (the paper uses GLM-5.2; a fully open-source reproduction would need to substitute a different model here, introducing an unavoidable point of divergence from the paper’s exact numbers).
• The exact trajectory-analysis prompt template used to query the external analyzer (the paper states this is given in “Figure 10” of its appendix; I did not independently verify this figure’s contents as part of writing this review, since access to the full appendix figure set was outside the scope of what was needed for the core method and theory verification this review focuses on).
• The same base backbone checkpoints (Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, Qwen3-1.7B-Instruct — all publicly available, which is a genuine reproducibility strength).
• The same offline rollout collection budget ($M=180$ tasks, $K_0=8$ rollouts each, 1,440 total trajectories) and the same three-epoch SFT training schedule.

For Stage 2 (Self-Evolving OPD):

• The complete hyperparameter table (Table 5), which the paper does provide in full: 150 policy updates, batch size 16 (ALFWorld/WebShop) or 128 (Search), rollout group $N=8$, learning rate $1\times10^{-6}$, $\epsilon_{\text{clip}}=0.2$, $\beta_{\text{opd}}=5.0$, $\lambda_{\text{opd}}=0.01$, $\beta_{\text{KL}}=0.01$, and the per-benchmark maximum interaction-step and prompt-length limits.
• The specific RL-stage trajectory-analysis and actor prompts (Figures 10 and 11 in the appendix, per the paper’s own cross-references — again, not independently re-verified as part of this review).
• The same evaluation protocol per benchmark: ALFWorld’s 140-task seen split and 134-task unseen split, WebShop’s 128 fixed test tasks, and the 51,713-question combined Search-QA evaluation set spanning seven datasets.

Compute environment: the paper states training was conducted on 8 NVIDIA A800 80GB GPUs — a substantial but not exotic compute budget, broadly reproducible by a well-resourced academic lab, though (as flagged in the Critical Assessment) the paper gives no wall-clock time or total GPU-hour figure for a full training run, so an independent reproducer has no direct basis for estimating how long a from-scratch run should take before concluding something has gone wrong.

Code availability: the abstract references a GitHub repository (github.com/jinyangwu/SEED per the paper’s citation list). As stated in the Reproducibility Notes above, I did not independently audit this repository’s completeness as part of writing this review — a reader planning an actual reproduction attempt should treat this review’s method-section description as a reading aid for understanding the paper’s mathematics and pipeline structure, not as a substitute for inspecting the authors’ own released implementation directly.

Quick-Reference Glossary

• POMDP: partially observable Markov decision process — the formal object modeling an agent that only observes partial information about a true, hidden environment state.
• GRPO: Group Relative Policy Optimization — an RL algorithm that samples a group of trajectories per task and normalizes their outcomes within the group to get a per-trajectory advantage, broadcast to all valid tokens in that trajectory.
• On-policy distillation (OPD): a distillation scheme where the tokens being scored under teacher/student contexts are ones the current (student) policy itself already sampled, rather than freshly generated by a separate teacher.
• Hindsight skill: a natural-language summary, generated after a trajectory completes, describing a reusable workflow (from success) or a corrective/avoidance rule (from failure).
• Confidence gate: a sigmoid function of the log-probability shift between skill-augmented and ordinary contexts, converting the shift into a soft $(0,1)$ weight for the distillation loss.
• Self-evolving loop: the property that the same, just-updated policy checkpoint serves both as the trajectory-collecting actor and the trajectory-analyzing hindsight generator at every training round, so neither role lags behind the other.
• Occupancy-matched target: a distillation target distribution constructed entirely from contexts and tokens the current policy actually visits (as opposed to off-policy data from a different or older policy).
• Stop-gradient (sg[·]): an operation that treats its argument as a constant during backpropagation, even though it was computed from parameters that do have gradients elsewhere — used throughout SEED to ensure the teacher branch and gate never receive gradient updates directly, only the student branch does.
• Value-alignment condition: the (empirically untested in the paper) requirement that the hindsight gate $w_k(c,v)$ be positively correlated with the true action value $Q_k(c,v)$, which Eq. 8 shows is necessary for the OPD update to actually improve expected returns rather than merely being well-defined.
• Analyzer staleness: the discrepancy between the gate values a current analyzer would produce versus an older, out-of-date analyzer, for the same underlying trajectory sample — bounded by Proposition 3 and driven to zero by SEED’s per-round refresh.
• Trust-region clipping: the PPO-style mechanism (via $\text{clip}(\rho, 1-\epsilon_{\text{clip}}, 1+\epsilon_{\text{clip}})$) that caps how much a single token’s importance ratio can influence the RL loss, preventing destructively large policy updates from data generated by a now-outdated frozen snapshot.

Additional Derivation Notes: The Softmax-Cross-Entropy Gradient Identity Used Throughout

Several of the derivations above (Proposition 1’s proof, Proposition 2’s Eq. 10, and my own numeric verifications) lean on one standard but easy-to-forget calculus identity: for a softmax distribution $p_z(v) = \exp(z(v))/\sum_u \exp(z(u))$, the derivative of the negative log-likelihood of a specific outcome $u$ with respect to the logit of a (possibly different) candidate $v$ is

$$\frac{\partial [-\log p_z(u)]}{\partial z(v)} = p_z(v) - \mathbb{1}\{u = v\}. \tag{Softmax-Grad}$$

This says: if you’re asking “how does increasing the logit of candidate $v$ change the loss of predicting $u$,” the answer is “it increases the loss by $p_z(v)$ if $v \ne u$ (competing probability mass is stolen from the correct answer $u$), but decreases it by $1 - p_z(v)$ if $v = u$ (directly reinforcing the correct answer).” Applying this identity to a weighted sum of such losses — as in Eq. 9’s $\mathcal{L}_{\text{opd},k,c}(z) = -\sum_v \pi_k(v)w_k(v)\log p_z(v)$, which is a weighted combination of NLL terms for every candidate $v$, not just one — and summing over all the outcomes gives exactly Eq. 10’s result: $\partial \mathcal{L}_{\text{opd},k,c}/\partial z(v) = \pi_\theta(v)\left[\sum_u \pi_k(u)w_k(u)\right] - \pi_k(v)w_k(v) = \pi_\theta(v) Z_k - \pi_k(v)w_k(v)$, which at $\theta=\theta_k$ (so $\pi_\theta = \pi_k$) becomes $\pi_k(v)[Z_k - w_k(v)]$, exactly as stated. I verified this derivation independently in my numeric checks above by comparing the finite-difference gradient against this closed form and finding agreement to 8 decimal places — the identity is not a paper-specific trick but standard softmax calculus, applied here to a skill-weighted target rather than a one-hot label.

A Closing Note on Reading This Paper Alongside SDAR and RLSD

For a reader planning to go deeper into this specific corner of the literature, my suggested reading order, based on working through all three papers’ mechanisms in this review, is: first RLSD (the simplest mechanically — a single modulation coefficient on an existing GRPO update, easiest to understand as a first exposure to “self-teacher” ideas), then SDAR (introduces the gating mechanism SEED directly borrows, in a context without the self-evolving analyzer machinery), then SEED itself (adds the self-evolving synchronization and the accompanying theoretical framework). Reading them in this order makes it much easier to isolate, concretely, which specific piece of machinery each successive paper is contributing, rather than trying to absorb all of SEED’s moving parts (hindsight-skill SFT, on-policy trajectory-skill generation, confidence gating, self-evolving refresh, joint RL+OPD optimization) simultaneously as one undifferentiated whole.

Conclusion

SEED makes a focused, well-motivated case that the mismatch between sparse, episode-level RL rewards and the fine-grained, token-level nature of LLM-agent policy learning can be substantially closed by turning completed trajectories into self-generated, self-evolving hindsight commentary, and distilling that commentary’s behavioral effect back into the ordinary policy at the individual-token level. Its theoretical contribution — three propositions formalizing exactly what “on-policy,” “dense,” and “self-evolving” buy you mathematically — gives genuine, checkable content behind the paper’s design choices, and my own numerical re-derivations confirm each proposition holds exactly as stated. Its empirical contribution is broad and consistent: gains over strong outcome-only RL and multiple prior self-distillation baselines, across three backbone scales, three qualitatively different agentic domains, and even a modest multimodal extension. The clearest remaining gaps are exactly the ones a careful reader should ask about — whether the value-alignment condition underlying the theory (Eq. 8) actually holds in practice, whether the shared hyperparameters across a 4x model-scale range are fair to the baseline, and what the true compute cost of running a dual actor/analyzer role really is relative to the reported sample-efficiency gains. None of these gaps invalidate the paper’s core claim, but they do mark the natural next experiments for anyone building directly on this work.