DPPO: Fixing PPO's Ratio Clipping by Constraining Divergence Instead of a Single Token's Probability

Note on scope: this review builds up the full theoretical chain (classical Kakade–Langford/TRPO theory through the paper’s finite-horizon re-derivation), the DPPO algorithm and its two divergence approximations with worked numeric examples, a section-by-section walk through the paper’s diagnostic experiments, and a critical assessment — it assumes comfort with basic probability and policy-gradient RL but builds every other prerequisite from scratch.

Review date: 2026-07-11 Review author: Zhongzhu Zhou Paper reviewed: Rethinking the Trust Region in LLM Reinforcement Learning Paper authors: Penghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang, Chao Du, Min Lin, Wee Sun Lee arXiv: 2602.04879 Status: ICML 2026 (PMLR 306), Sea AI Lab & National University of Singapore

Short Answer

Every RL recipe used to fine-tune large language models today — PPO, GRPO, and their many descendants — inherits a trust-region mechanism designed in 2017 for small, dense-action classical-control environments and never re-derived for the enormous, long-tailed vocabularies that LLMs sample from. This paper argues, and then carefully demonstrates, that this inherited mechanism is broken in a specific and fixable way: PPO’s clipping decision is driven by the probability ratio of a single sampled token, which is a wildly noisy, single-sample Monte Carlo estimate of the thing the trust region is actually supposed to control — the total-variation (or KL) divergence between the old and new token distributions. Because LLM vocabularies are enormous and long-tailed, that noisy proxy fails in two directions at once: it aggressively clips updates to rare tokens even when the actual probability mass moved is tiny, slowing down exploration and reasoning-token learning, while it lets updates to dominant tokens slide through un-clipped even when they move a large amount of probability mass, risking catastrophic, destabilizing shifts. The paper’s fix, Divergence Proximal Policy Optimization (DPPO), keeps PPO’s simple masking structure but replaces the ratio with a direct, cheap estimate of the actual divergence between the rollout policy and the training policy, using two lightweight approximations (Binary and Top-K) that make this tractable at the scale of a 150k-token vocabulary. Along the way, the authors re-derive the classical Kakade–Langford policy-improvement bound for the finite-horizon, sparse-reward, sequence-level setting that actually describes LLM RL (rather than the infinite-horizon discounted MDP the original theorem was built for), and they run a battery of diagnostic experiments that pin down exactly which policy updates cause the catastrophic collapses everyone doing RL post-training has seen — a small fraction (often well under 1%) of updates on negatively rewarded tokens where the policy has shifted far from the rollout distribution. DPPO outperforms GRPO with Clip-Higher and CISPO across five large-scale configurations (dense and MoE, with and without rollout-router-replay, verifiable-reward and RLHF-style rewards), while removing the need for an expensive recomputed reference policy, cutting training cost by roughly 25%.

Key Takeaways

• PPO’s ratio-clipping mechanism was inherited from classical RL and was never re-validated for LLM-scale, long-tailed vocabularies; its trust-region safeguard is a single-sample estimate of TV divergence, and single samples of rare-token probability ratios are extremely high-variance.
• Structural flaw, illustrated with numbers: a token moving from probability $10^{-4}$ to $10^{-2}$ has ratio $100$ (heavily clipped) but moves only $\approx 0.0099$ probability mass; a dominant token moving from $0.99$ to $0.80$ has ratio $\approx 0.808$ (barely clipped, if at all) but moves $0.19$ probability mass — 19x more actual distributional shift, yet the less constrained update.
• DPPO replaces the ratio-based mask with a mask conditioned on an actual divergence estimate $D(\mu(\cdot|s_t) \| \pi(\cdot|s_t))$, computed cheaply via a Binary approximation (collapse the vocabulary to “sampled token” vs. “everything else”) or a Top-K approximation (track the $K$ highest-probability tokens under the rollout policy plus an aggregated “other” bucket).
• The paper re-derives the Kakade–Langford performance-difference identity and policy-improvement bound for the finite-horizon ($T$), undiscounted ($\gamma=1$), sequence-reward setting that actually describes autoregressive LLM generation — the textbook infinite-horizon-discounted bound is literally undefined ($\frac{1}{1-\gamma}\to\infty$) in this regime.
• A dedicated ablation study answers three concrete empirical questions: (1) yes, a trust region remains necessary even at learning rates as low as $10^{-6}$; (2) the trust region must be anchored to the original rollout policy $\mu_{\theta'}$, not a recomputed on-policy distribution $\pi_{\theta'}$ — anchoring to the recomputed distribution reintroduces the exact collapse a trust region is meant to prevent; (3) fewer than 0.5% of updates — specifically, large negative-advantage updates that push a token’s probability down by more than 0.5 — are responsible for essentially all observed training collapses.
• On the efficiency side, relaxing PPO’s clip only for low-probability tokens ($\mu(y_t|s_t) < \alpha$) substantially speeds up learning without hurting stability, and relaxing the lower clip bound matters more than the upper bound (opposite of the popular “Clip-Higher” intuition) — DPPO’s asymmetric masking captures this automatically because it never blocks moves toward the rollout distribution.
• Across five large-scale training configurations (MoE base ±rollout-router-replay, MoE “thinking” model, dense 8B base, MoE+LoRA) and two RLHF alignment settings, DPPO-Binary matches or beats GRPO-ClipHigher and CISPO, in several cases outperforming even the R3-stabilized MoE baseline while using less compute (no recomputed reference policy needed).
• Binary and Top-K approximations perform similarly in the ablation, meaning the cheapest possible approximation — collapsing the vocabulary to a 2-outcome Bernoulli around the sampled token — already captures most of the achievable benefit; there is little practical reason to pay for the richer Top-K variant in production.
• The paper’s Appendix D qualitative analysis shows the specific tokens PPO over-clips are not random rare tokens — they are disproportionately numerical/mathematical symbols and reasoning-discourse connectives (Wait, Since, Thus), meaning ratio-clipping’s mispricing is silently degrading the parts of a chain-of-thought response that carry logical structure.
• Generalization checks beyond AIME-style math RL — a different base model family (OctoThinker), abstract reasoning and induction tasks (Arc1D, Acre), and a genuinely multi-turn environment (Sudoku) — all show the same efficiency advantage for divergence-based masking over ratio-based masking, holding the rest of the training recipe fixed.

A Final Sanity Check on the Central Claim

Before closing, it is worth restating the central empirical claim in the most falsifiable form possible, since that is the strongest test of whether this review has actually understood the paper: if you took any existing PPO/GRPO-based LLM RL pipeline, changed nothing except (a) anchoring the trust region to the rollout distribution instead of a recomputed one, and (b) replacing the ratio-threshold mask with a Binary-divergence-threshold mask, the paper’s evidence predicts you should observe faster reward growth, no catastrophic collapse across a wide range of model scales and task types, and roughly 25% lower training compute from removing the recomputation pass — all without touching the advantage estimator, the reward function, or any other part of the pipeline. That is a strong, specific, and testable prediction, which is exactly the kind of claim a methods paper should be making.

Model Scale Summary Across All Experiments

For quick reference, here is every base model used somewhere in the paper’s experiments, spanning roughly three orders of magnitude in parameter count and both dense and Mixture-of-Experts architectures:

Model	Type	Used in
DeepSeek-R1-Distill-Qwen-1.5B	dense	RQ1–RQ3 stability diagnostics (Section 5)
Qwen3-1.7B-Base	dense	efficiency analysis (Section 6), abstract-reasoning/induction/multi-turn generalization (Appendix F.5)
Qwen3-4B-Base	dense	clipped-token qualitative analysis (Appendix D)
Qwen3-4B-Instruct-2507	dense	RLHF alignment experiment on HH-RLHF (Section 7.2)
Qwen3-8B-Base	dense	broader evaluation “Dense Base” configuration (Section 7.1)
Qwen3-30B-A3B-Base / Qwen3-30B-A3B	MoE	broader evaluation “MoE Base,” “MoE Base + R3,” “MoE Thinking,” “MoE + LoRA” configurations; hyperparameter sensitivity (Appendix F.4)
Gemma-2-9B-It	dense	RLHF alignment experiment on UltraFeedback (Section 7.2)
OctoThinker-3B-Hybrid-Base	dense	cross-model-family generalization check (Appendix F.5)
Skywork-Reward-Llama-3.1-8B	dense (reward model only, not RL-trained)	reward model for both RLHF experiments

Quick Reference: When Does This Paper’s Fix Matter Most?

A short decision-oriented summary of when the gap this paper identifies is most likely to bite, versus when it may matter less, based on the mechanisms discussed throughout this review:

Situation	Relevance of this paper’s fix
Long chain-of-thought RL training (thousands of tokens per response)	high — long sequences accumulate more opportunities for the rare “bad update” pattern (RQ3), and the average-divergence bound (T5) is specifically tighter here
Large, long-tailed vocabularies (100k+ tokens, many rare/technical/multilingual tokens)	high — this is the exact regime where ratio-based clipping’s mispricing is most pronounced
MoE models with routing-induced training-inference mismatch	high — DPPO shown to help even beyond what R3 alone provides
Small action-space classical RL (e.g., low-dimensional continuous control)	low — this is closer to the regime PPO’s original ratio-clip approximation was designed for and validated on
Low-precision, high-throughput inference where some training-inference mismatch is unavoidable	high — the paper frames this as exactly where a robust algorithmic layer like DPPO is most valuable
Already using an engine-alignment solution with near-zero training-inference mismatch	lower, but not zero — RQ1 shows a trust region still matters even when mismatch is small, since it can still accumulate over many steps

Closing Thought

The most memorable idea in this paper, stripped of all notation, is almost embarrassingly simple once stated plainly: if you want to know whether an update is safe, measure the thing you actually care about (how far the distribution moved), not a cheap proxy for it (how far one random sample happened to land). That PPO’s entire LLM-era deployment quietly relied on the proxy, for years, before anyone re-derived the theory carefully enough to notice, is itself a useful reminder that widely-adopted defaults are worth periodically re-examining rather than assumed correct by virtue of ubiquity.

Where to Find the Code

The authors link an implementation at github.com/sail-sg/Stable-RL. Since this review emphasizes that DPPO’s core change (the mask in Equation D2 plus the Binary or Top-K divergence estimate) is a small, self-contained modification to an existing PPO/GRPO training loop rather than a new training system, the most direct way to verify anything in this review is to compare that mask implementation directly against the ratio-based clip in whatever training framework (VeRL, Oat, or another) you already use, rather than needing to adopt the authors’ full stack wholesale.

Acronym Glossary

Given the density of algorithm names in this space, a quick decoder:

Acronym	Expansion
MDP	Markov Decision Process
TRPO	Trust Region Policy Optimization
PPO	Proximal Policy Optimization
GRPO	Group Relative Policy Optimization
DPPO	Divergence Proximal Policy Optimization (this paper’s method)
DAPO	an open-source large-scale LLM RL system that introduced the Clip-Higher trick
CISPO	a truncated-importance-sampling-based critic-free RL method
RLOO	REINFORCE Leave-One-Out
VAPO	Value-Augmented Proximal Policy Optimization
GSPO	Group Sequence Policy Optimization
RLHF	Reinforcement Learning from Human Feedback
TIS	Truncated Importance Sampling
KL	Kullback–Leibler (divergence)
TV	Total Variation (divergence)
MoE	Mixture of Experts
R3	Rollout Router Replay (MoE expert-routing stabilization technique)
LoRA	Low-Rank Adaptation
MM (algorithm)	Minorize-Maximization
AIME	American Invitational Mathematics Examination (used here as an RL evaluation benchmark)
DAPO-Math / MATH	curated math-reasoning training datasets used across the paper’s scaling and stability experiments
UltraFeedback / HH-RLHF	public preference/alignment datasets used for the RLHF-style experiments (Section 7.2)

Notation Reference

This paper carries a lot of symbols across its theory and method sections; I collect the recurring ones here as a lookup table, since I will refer back to these repeatedly.

Symbol	Meaning
$\mu$, $\pi$	behavior/rollout policy and target/training policy, respectively
$\mu_{\theta'}$, $\pi_\theta$	rollout policy at parameters $\theta'$ (data-generating), training policy at current parameters $\theta$
$s_t = (x, y_1,\dots,y_{t-1})$	state at step $t$: the prompt plus tokens generated so far
$y = (y_1,\dots,y_T)$	full generated response, length $T$
$R(y)$	scalar terminal reward for the full response
$J(\pi) = \mathbb{E}_{y\sim\pi}[R(y)]$	expected sequence-level reward under policy $\pi$
$\hat{A}_t$	estimated (GRPO-style, group-relative) advantage at step $t$
$r_t = \pi(y_t	s_t)/\mu(y_t
$D_{TV}(\mu\|\pi)$, $D_{KL}(\mu\|\pi)$	total-variation and KL divergence between two token distributions
$D_{TV}^{\max}$	worst-case (max over states) TV divergence
$\bar{D}_{TV}$	average, trajectory-summed per-token TV divergence actually realized
$\delta$	divergence threshold hyperparameter (DPPO’s analogue of PPO’s $\epsilon$)
$\epsilon$, $\epsilon_{low}$, $\epsilon_{high}$	PPO/GRPO clip bounds (symmetric or asymmetric)
$M_t$	binary mask (0 = block gradient, 1 = pass gradient) at step $t$
$A’_t = \text{TopK}(\mu(\cdot	s_t),K)\cup{a_t}$
$\xi$	maximum absolute reward magnitude, $\max_y
$L_\mu(\pi)$, $L'_\mu(\pi)$	trust-region surrogate objectives (classical discounted-MDP form, and the paper’s finite-horizon form)
$\Delta(\mu,\pi)$	exact residual/error term between the surrogate and the true objective (Theorem 3.1)

Equation Index

Given how many labeled equations this review builds up, here is a flat index of all of them with a one-line pointer back to what each represents — useful for jumping around rather than reading linearly.

Tag	What it is
P1	classical discounted RL objective $\eta(\pi)$
P2	Kakade–Langford exact performance-difference identity
P3	TRPO’s lower-bound surrogate with the max-TV penalty
P4	TRPO’s constrained-optimization trust-region template
P5	PPO’s clipped surrogate objective
P6	GRPO’s group-relative advantage estimator
T1	this paper’s exact finite-horizon performance-difference identity
T2	the finite-horizon surrogate $L'_\mu(\pi)$
T3	the exact residual error term $\Delta(\mu,\pi)$
PF1	full-sequence importance ratio as a product of per-token ratios
PF2	the telescoped, per-token sum form of the exact identity
T4	policy-improvement bound using worst-case (max) divergence
T5	policy-improvement bound using average realized divergence
T6	this paper’s constrained-optimization template (finite-horizon analogue of P4)
D1	the DPPO objective $L_\mu^{DPPO}(\pi)$
D2	the DPPO mask $M_t^{DPPO}$
D3	Binary approximation of TV divergence
D4	Binary approximation of KL divergence
D5	Top-K reduced categorical distribution construction
D6	Top-K approximation of TV and KL divergence
U1	the unified policy-gradient template covering all baselines and DPPO
E1	the RQ3 minimal “bad update” mask, Equation from paper Section 5.3

Prerequisites: What You Need to Know First

This paper sits at the intersection of classical trust-region policy optimization theory (TRPO/PPO, 2015-2017) and the modern LLM RL-fine-tuning stack (GRPO-style critic-free training, 2023-2026). To follow the derivations you need six pieces of background, which I build up in order: the MDP formalism and policy gradients, the Kakade–Langford performance-difference identity, how TRPO turns that identity into a trust-region constraint, how PPO approximates TRPO cheaply via ratio clipping, how GRPO removes the critic for LLM fine-tuning, and finally how LLM generation itself gets cast as an MDP (which is where the paper’s real contribution begins).

From MDPs to Policy Gradients

A Markov Decision Process is a tuple $\mathcal{M} = (\mathcal{S}, \mathcal{A}, P, r, \rho_0, \gamma)$: a state space, an action space, transition dynamics $P(s'|s,a)$, a reward function $r(s,a)$, an initial state distribution $\rho_0$, and a discount factor $\gamma \in [0,1]$. A stochastic policy $\pi(a|s)$ generates trajectories $\tau = (s_0, a_0, r_0, s_1, a_1, r_1, \dots)$ by sampling $a_t \sim \pi(\cdot|s_t)$ and transitioning $s_{t+1}\sim P(\cdot|s_t,a_t)$. The RL objective is to maximize expected discounted return

$$\eta(\pi) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^t r_t\right]. \tag{P1}$$

Two auxiliary quantities matter for everything that follows: the state-value function $V^\pi(s) = \mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^\infty \gamma^t r_t \mid s_0=s\right]$, and the advantage function $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$, which measures how much better action $a$ is than the policy’s average behavior at state $s$. Advantage is the natural currency of policy-gradient methods: a positive advantage means “do more of this,” a negative advantage means “do less of this.”

The Kakade–Langford Identity: Comparing Two Different Policies

The central theoretical tool the whole trust-region literature is built on is the policy performance difference theorem (Kakade & Langford, 2002). For any two policies — a target policy $\pi$ we want to evaluate/optimize, and a behavior policy $\mu$ that actually generated the data — their expected returns are related exactly by

$$\eta(\pi) - \eta(\mu) = \frac{1}{1-\gamma}\,\mathbb{E}_{s\sim\rho^\pi,\, a\sim\pi(\cdot|s)}\left[A^\mu(s,a)\right], \tag{P2}$$

where $\rho^\pi(s) = (1-\gamma)\sum_{t=0}^\infty \gamma^t \Pr(s_t=s\mid\pi)$ is the discounted state-visitation distribution induced by $\pi$. Read this carefully: it says the improvement of $\pi$ over $\mu$ equals the expected advantage of $\pi$‘s own actions, measured under $\mu$‘s value function, but averaged over states visited by $\pi$, not $\mu$. This is both exact and useless for direct optimization, because we only have samples from $\mu$ (the policy that generated our data), not from $\pi$ (the policy we’re trying to evaluate) — we cannot cheaply sample $\rho^\pi$.

TRPO: Turning an Intractable Identity into an Optimizable Bound

Schulman et al. (2015) resolve this by proving a lower bound on Equation P2 that swaps $\rho^\pi$ for the tractable $\rho^\mu$, at the cost of an extra penalty term measuring how far $\pi$ has drifted from $\mu$:

$$\eta(\pi)-\eta(\mu) \;\ge\; \underbrace{\frac{1}{1-\gamma}\,\mathbb{E}_{s\sim\rho^\mu,\,a\sim\mu(\cdot|s)}\left[\frac{\pi(a|s)}{\mu(a|s)}A^\mu(s,a)\right]}_{=:L_\mu(\pi),\ \text{the surrogate objective}} \;-\; \frac{2\xi\gamma}{(1-\gamma)^2}\,D_{TV}^{\max}(\mu\|\pi)^2, \tag{P3}$$

where $\xi=\max_{s,a}|A^\mu(s,a)|$ and $D_{TV}^{\max}(\mu\|\pi)=\max_s D_{TV}\big(\mu(\cdot|s)\|\pi(\cdot|s)\big)$ is the worst-case per-state total-variation divergence. Here is the intuition behind each piece: $L_\mu(\pi)$ is an importance-weighted estimate of the true objective that you can compute from $\mu$‘s samples (this is exactly where the familiar “probability ratio times advantage” term comes from) — its value and gradient match the true $\eta(\pi)-\eta(\mu)$ exactly at $\pi=\mu$, but it drifts away as $\pi$ moves further from $\mu$. The penalty term quantifies exactly how much it can drift, as a function of the worst-case divergence between the two policies. So Equation P3 says: if you maximize $L_\mu(\pi)$ while keeping $D_{TV}^{\max}(\mu\|\pi)$ small, you are guaranteed a lower bound on real improvement — and because the bound touches the true objective at $\pi=\mu$, iteratively maximizing it is a Minorize-Maximization (MM) procedure that provably never decreases $\eta$. This motivates the constrained problem

$$\max_\pi\ L_\mu(\pi)\quad\text{s.t.}\quad D_{TV}^{\max}(\mu\|\pi)\le \delta, \tag{P4}$$

which is what “trust region” means formally: a hard constraint on the divergence, not a soft heuristic.

PPO: Approximating the Trust Region With a Cheap Clip

TRPO’s constrained optimization in Equation P4 requires second-order information (a KL-constrained natural-gradient step) and does not scale well to billion-parameter networks. PPO (Schulman et al., 2017) replaces the hard constraint with a much cheaper first-order surrogate: clip the per-token importance ratio $r_t = \pi(y_t|s_t)/\mu(y_t|s_t)$ directly in the objective,

$$L^{PPO}_\mu(\pi) = \mathbb{E}_{y\sim\mu}\left[\sum_{t=1}^{|y|}\min\big(r_t \hat{A}_t,\ \operatorname{clip}(r_t, 1-\epsilon, 1+\epsilon)\,\hat{A}_t\big)\right]. \tag{P5}$$

The connection to the formal trust region is that the token-level TV divergence at a single state can itself be written in terms of the ratio: $D_{TV}\big(\mu(\cdot|s_t)\|\pi(\cdot|s_t)\big) = \frac{1}{2}\mathbb{E}_{y_t\sim\mu}|r_t - 1|$. So PPO’s clip condition $|r_t - 1|\le\epsilon$ can be read as constraining a single-sample Monte Carlo estimate of this expectation — using exactly one draw ($y_t$, the token that was actually sampled) to estimate an expectation over the entire vocabulary. This is the crux the whole paper hinges on, so hold onto it: PPO enforces its trust region on a noisy point-estimate of divergence, not divergence itself.

GRPO: Removing the Critic for LLM Fine-Tuning

Training a value function $V^\pi(s)$ for an LLM is expensive (another full-size network) and often noisy given sparse, sequence-level rewards. GRPO (Shao et al., 2024) and related critic-free methods sidestep this by generating a group of $G$ responses to the same prompt and using the group-relative reward as a variance-reduced advantage estimate:

$$\hat{A} = R(y) - \frac{1}{G}\sum_{i=1}^G R(y_i). \tag{P6}$$

This paper treats PPO (Equation P5) and GRPO as instances of the same underlying algorithm, differing only in how $\hat{A}$ is computed — the paper’s target, ratio-clipping, is shared by both, so DPPO’s fix applies to GRPO exactly as it applies to vanilla PPO.

A concrete numeric example of Equation P6. Suppose a prompt generates a group of $G=4$ responses with rewards $R(y_1)=1$ (correct answer), $R(y_2)=0$ (incorrect), $R(y_3)=1$ (correct), $R(y_4)=0$ (incorrect) — a typical binary-verifiable-reward setting. The group mean is $\frac{1}{4}(1+0+1+0)=0.5$, so the advantages are $\hat{A}(y_1)=1-0.5=0.5$, $\hat{A}(y_2)=0-0.5=-0.5$, $\hat{A}(y_3)=0.5$, $\hat{A}(y_4)=-0.5$. Every token in a correct response gets the same positive advantage $0.5$ regardless of which token it is or how it contributed to correctness; every token in an incorrect response gets the same negative advantage $-0.5$. This is the sense in which GRPO’s advantage is coarse: it cannot distinguish “this specific token was the crucial insight” from “this token was a generic connective,” it only knows “this token belongs to an overall-correct (or overall-incorrect) response.” That coarseness is exactly why the masking mechanism applied on top of $\hat{A}_t$ matters so much — since the advantage signal itself cannot discriminate which tokens deserve more or less trust, the trust-region mask (PPO’s ratio-based one, or DPPO’s divergence-based one) is the only mechanism left that can modulate how strongly any individual token’s gradient is actually applied, which is precisely why getting that mechanism right (this paper’s whole argument) has an outsized effect on which tokens end up learning efficiently.

KL vs. Total Variation Divergence

Two divergence measures recur throughout: KL divergence $D_{KL}(\mu\|\pi) = \sum_a \mu(a)\log\frac{\mu(a)}{\pi(a)}$, and total variation $D_{TV}(\mu\|\pi) = \frac{1}{2}\sum_a|\mu(a)-\pi(a)|$. They are related by Pinsker’s inequality, $D_{TV}(\mu\|\pi)^2 \le \frac{1}{2}D_{KL}(\mu\|\pi)$, which is why a bound proven for one divergence transfers (with a constant-factor loss of tightness) to the other — this is why the paper’s theory is stated with TV divergence but its algorithm supports either TV or KL interchangeably.

The LLM Generation Process as a Finite-Horizon MDP

Here is where the classical machinery starts to strain. Given a prompt $x$, an LLM policy $\pi$ generates a response $y=(y_1,\dots,y_T)$ token-by-token, where at each step the “state” $s_t = (x, y_1,\dots,y_{t-1})$ is the prompt plus everything generated so far, and the response probability factorizes as $\pi(y|x) = \prod_{t=1}^T \pi(y_t|s_t)$. Critically, this is a finite-horizon, undiscounted ($\gamma=1$) process with a single terminal reward $R(y,x)$ assigned only after the whole response completes — there is no per-step reward, no infinite trajectory, and (because $\gamma=1$) the $\frac{1}{1-\gamma}$ factor that appears throughout Equations P2–P5 is literally undefined (it blows up to infinity). This means the classical TRPO/PPO bound (Equation P3) does not merely need re-tuning for LLMs — it is mathematically ill-posed in this regime, and needs to be re-derived from scratch for finite-horizon, sparse-terminal-reward sequences. That re-derivation (Theorems 3.1 and 3.2 below) is the paper’s first real theoretical contribution, and everything about DPPO’s design follows from it.

Training-Inference Mismatch: The Practical Trigger

One more piece of context before the method: in practice, LLM RL pipelines use a fast inference engine (e.g., vLLM) to sample rollouts and a separate training engine to compute gradients. Even with byte-identical parameters $\theta$, these two engines can produce slightly different token probabilities $\mu_{\theta'}\neq\pi_\theta$ due to numerical precision, kernel implementation differences, and batching effects — a phenomenon called training-inference mismatch. Practical systems also reuse one batch of rollouts for several gradient minibatch updates to improve throughput, which further widens the gap between the policy that generated the data and the policy currently being trained. Both effects mean $\mu$ and $\pi$ are never exactly identical even at the start of a training step, which is exactly the setting the trust-region machinery above is meant to handle — and exactly where its LLM-specific failure mode (ratio clipping being a bad divergence proxy) becomes practically visible as training collapse.

A Brief History: From TRPO (2015) to DPPO (2026)

Situating this paper on a timeline makes clear how long the gap it identifies has quietly existed, since the underlying mismatch between “the trust region we want” and “the trust region PPO’s clip actually enforces” long predates the LLM era:

• 2015 — TRPO (Schulman et al.): proves the classical policy-improvement bound (Equation P3) and the constrained-optimization trust-region template (Equation P4), for general discounted MDPs with typically small, often discrete-and-low-cardinality action spaces (e.g., Atari, MuJoCo control).
• 2017 — PPO (Schulman et al.): replaces TRPO’s expensive second-order constrained optimization with cheap first-order ratio clipping (Equation P5), trading exactness for scalability — a reasonable trade in classical RL’s small-action-space regime, where a single-sample ratio is a much less noisy proxy for divergence than it is in a 100k+-token vocabulary.
• 2022–2023 — LLM RL takes off: InstructGPT-style RLHF and, later, GRPO (2024) adopt PPO’s clipping machinery essentially unchanged, now applied to token-level ratios over enormous vocabularies — the paper’s core claim is that nobody re-validated whether the single-sample-ratio proxy was still a good approximation once the action space grew by many orders of magnitude.
• 2025 — symptom-patching heuristics appear: Clip-Higher (DAPO) and CISPO each independently notice that low-probability “exploration”/“reasoning” tokens are disproportionately clipped, and each proposes a one-sided patch (widen the upper bound; truncate instead of clip) without diagnosing why ratio clipping mis-prices these tokens in the first place.
• 2025 — training-inference mismatch is named explicitly: separate work (Yao et al.; Qi et al.) identifies and characterizes the training-inference mismatch phenomenon this paper’s method must also contend with, spurring engineering mitigations (TIS, engine-alignment efforts) that this paper shows are incomplete or, in TIS’s case, sometimes actively counterproductive.
• 2026 — this paper: re-derives the theoretical foundation from scratch for the actual regime LLM RL operates in (finite-horizon, undiscounted, terminal-reward), diagnoses the root cause (ratio is a bad divergence proxy, not merely “sometimes too strict”), and replaces the proxy rather than patching around it.

Read this way, the paper’s contribution is less “a clever new trick” than “someone finally went back and checked whether the load-bearing assumption from 2017 still held in 2026” — and found that it quietly did not, in a way that had been silently costing both training stability and training efficiency the entire time.

The Core Diagnosis: Why Ratio Clipping Is the Wrong Proxy

flowchart TB
    subgraph PPO["PPO: ratio-based masking (heuristic proxy)"]
        A1["sample token a_t ~ mu(.|s_t)"] --> A2["compute ratio r_t = pi(a_t|s_t) / mu(a_t|s_t)"]
        A2 --> A3{"is r_t outside [1-eps, 1+eps]?"}
        A3 -- yes --> A4["clip: block gradient on this token"]
        A3 -- no --> A5["pass gradient through unmodified"]
    end
    subgraph DPPO["DPPO: divergence-based masking (direct estimate)"]
        B1["sample token a_t ~ mu(.|s_t)"] --> B2["estimate D = divergence(mu(.|s_t), pi(.|s_t))"]
        B2 --> B3{"is D above threshold delta AND update moving away from mu?"}
        B3 -- yes --> B4["mask: block gradient on this token"]
        B3 -- no --> B5["pass gradient through unmodified"]
    end

Figure 1 (paper Fig. 1, architecture overview): PPO decides whether to clip using only the single sampled token’s probability ratio — a one-sample estimate of the true distributional shift. DPPO instead estimates the actual divergence between the full rollout and training distributions at that state, and masks only when the estimated divergence exceeds a threshold and the update is moving further away from the rollout policy.

The paper motivates this redesign with a clean numeric example worth internalizing, because it explains why single-token ratios are such bad divergence estimates for LLMs specifically (as opposed to, say, a robotics MDP with 8 continuous action dimensions). Consider a fixed state $s$ and two tokens:

| Token | $\mu(a|s)$ (rollout) | $\pi(a|s)$ (training) | Ratio $r=\pi/\mu$ | Probability mass moved | |---|---|---|---|---| | $a_{low}$ (rare token) | $10^{-4}$ | $10^{-2}$ | $100$ | $\approx 0.0099$ | | $a_{high}$ (dominant token) | $0.99$ | $0.80$ | $\approx 0.808$ | $0.19$ |

Figure 2 (comparison figure, paper Section 4.2): a rare token’s probability moving from $10^{-4}$ to $10^{-2}$ produces a ratio of 100 — deep inside PPO’s clipping zone — despite moving under 1% of probability mass. A dominant token’s probability moving from $0.99$ down to $0.80$ produces a ratio of only $0.808$ — likely inside a typical clip range $[0.8, 1.2]$ — despite moving 19x more probability mass. PPO’s clip fires on the wrong one.

With a typical clip range $\epsilon=0.2$ (i.e., $r\in[0.8,1.2]$), PPO would clip $a_{low}$‘s update hard — even though its actual contribution to distributional shift is negligible — while letting $a_{high}$‘s update through mostly unconstrained, even though it is the one that actually risks moving the policy far from where it started. This is not a corner case: LLM vocabularies are long-tailed by construction (subword tokenizers, rare technical terms, multilingual tokens), so a meaningful fraction of every rollout’s tokens sit in the $a_{low}$ regime, meaning this mispricing happens continuously, not occasionally. The paper’s Figure 2 (reproduced conceptually here) shows this empirically on Qwen3-30B-A3B-Base: measured directly, the ratio is highly volatile for low-probability tokens while the TV divergence contribution stays smooth and small — confirming that ratio is the noisy quantity and TV divergence is the stable, well-behaved one. Two prior heuristics (Clip-Higher, which manually raises the upper clip bound, and CISPO, which lets gradients through regardless of divergence) each patch one symptom of this mismatch without touching its root cause; the paper’s stated goal is to remove the root cause rather than compensate for its symptoms.

Related Heuristics: Clip-Higher and CISPO Treat the Symptom, Not the Disease

It is worth being precise about why two well-known prior patches do not solve the problem the paper identifies, because both correctly notice the symptom (low-probability tokens getting clipped too aggressively) without fixing the underlying cause (the ratio is a bad divergence proxy). Clip-Higher (Yu et al., 2025) keeps PPO’s ratio-based mechanism entirely intact and simply widens the upper clip bound, $\epsilon_{high}$, so that ratios moderately above 1 survive un-clipped. This helps in the specific case where a token’s ratio happens to land just outside the old, tighter bound, but it does nothing for the deeper problem: the decision of whether to clip is still driven by the single-sample ratio, so a rare token whose ratio is 100 (as in the $a_{low}$ example above) is still clipped regardless of how wide $\epsilon_{high}$ is set, because 100 is far outside any reasonable clip range. It is a recalibration of the same broken instrument, not a new instrument. CISPO (Chen et al., 2025) takes the opposite tack: instead of clipping, it lets the (possibly truncated) importance-weighted gradient through regardless of how large the divergence implied by the ratio actually is. This solves the low-probability-token-gets-under-trained problem directly — nothing is ever blocked — but at the cost of removing the trust-region safeguard altogether, which is exactly why CISPO appears in the RQ1 stability experiment above as one of the methods whose training-inference mismatch grows unboundedly and eventually collapses. DPPO’s contribution is to notice that both patches are addressing the same underlying misdiagnosis (treating ratio-based clipping as if it were divergence-based clipping) from opposite directions, and to instead fix the diagnosis: keep a real trust region, but measure the thing the trust region is supposed to measure.

Theory: Trust Regions Rebuilt for the LLM Regime

Theorem 3.1 — An Exact Performance Difference Identity for Finite-Horizon Sequences

Because $\gamma=1$ breaks the classical identity (Equation P2), the paper derives a fresh exact identity for the finite-horizon, terminal-reward setting. Writing $J(\pi)=\mathbb{E}_{y\sim\pi}[R(y)]$ for the expected sequence-level reward, the result is:

$$J(\pi) - J(\mu) = L'_\mu(\pi) - \Delta(\mu,\pi), \tag{T1}$$

where the tractable surrogate (computable from $\mu$‘s rollouts) is

$$L'_\mu(\pi) = \mathbb{E}_{y\sim\mu}\left[R(y)\sum_{t=1}^{|y|}\left(\frac{\pi(y_t|s_t)}{\mu(y_t|s_t)}-1\right)\right], \tag{T2}$$

and the exact residual error term is

$$\Delta(\mu,\pi) = \mathbb{E}_{y\sim\mu}\left[R(y)\sum_{t=1}^{|y|}\left(\frac{\pi(y_t|s_t)}{\mu(y_t|s_t)}-1\right)\left(1-\prod_{j=t+1}^{T}\frac{\pi(y_j|s_j)}{\mu(y_j|s_j)}\right)\right]. \tag{T3}$$

Why this identity holds, and how to read it. $L'_\mu(\pi)$ is a first-order approximation: it estimates how much reward changes if you perturb the probability of each individual token $y_t$ by its importance ratio, treating all tokens’ contributions as if they were independent and additive — which is exactly the same kind of importance-weighted linearization used in the classical TRPO surrogate (Equation P3), just re-derived for a sequence with one terminal reward rather than per-step rewards. $\Delta(\mu,\pi)$ captures everything that first-order view misses: the product term $\prod_{j=t+1}^T \frac{\pi(y_j|s_j)}{\mu(y_j|s_j)}$ measures how much the rest of the sequence (tokens after position $t$) has also shifted, and the term is exactly zero when $\pi=\mu$ everywhere (both factors collapse to zero), which is why $L'_\mu$ exactly equals $J(\pi)-J(\mu)$ at that point and the identity is not just a bound but an equality, no approximation involved. The intuition for why a sequence-level reward needs this correction term at all, whereas the classical per-step-reward case (Equation P2) does not, is that in the classical case the value function $V^\mu$ already “prices in” everything that happens after time $t$; here, with only a terminal reward and no intermediate value function, the algebra has to explicitly track how the ratio of later tokens compounds with the ratio at the current token — hence the product term.

Proof Sketch: Where the Telescoping Product in Theorem 3.1 Comes From

It is worth unpacking why a product of future ratios appears in the error term $\Delta(\mu,\pi)$, since it looks unmotivated on first read. Start from the exact ratio between the two policies’ probabilities of generating the same full response $y$:

$$\frac{\pi(y|x)}{\mu(y|x)} = \prod_{j=1}^{T}\frac{\pi(y_j|s_j)}{\mu(y_j|s_j)}. \tag{PF1}$$

The exact importance-sampling identity for the objective is $J(\pi) = \mathbb{E}_{y\sim\mu}\left[\frac{\pi(y|x)}{\mu(y|x)}R(y)\right]$ — swap the sampling distribution to $\mu$ and reweight by the full-sequence ratio, and the expectation is unchanged because this is just the definition of importance sampling, no approximation yet. Subtracting $J(\mu) = \mathbb{E}_{y\sim\mu}[R(y)]$ from both sides gives $J(\pi)-J(\mu) = \mathbb{E}_{y\sim\mu}\left[R(y)\left(\prod_{j=1}^T \frac{\pi(y_j|s_j)}{\mu(y_j|s_j)} - 1\right)\right]$, an exact identity in terms of one big product over the whole sequence. The key algebraic step the paper takes is to telescope this one big product-minus-one term into a sum of per-token contributions, using the identity $\prod_{j=1}^T c_j - 1 = \sum_{t=1}^T (c_t - 1)\prod_{j=t+1}^T c_j$ for any sequence of scalars $c_j$ (this is a standard telescoping trick: check it by induction on $T$, or by noting that each term in the sum “un-does” one more factor of the product moving right to left). Applying this with $c_t = \pi(y_t|s_t)/\mu(y_t|s_t)$ gives exactly

$$J(\pi)-J(\mu) = \mathbb{E}_{y\sim\mu}\left[R(y)\sum_{t=1}^{T}\left(\frac{\pi(y_t|s_t)}{\mu(y_t|s_t)}-1\right)\prod_{j=t+1}^{T}\frac{\pi(y_j|s_j)}{\mu(y_j|s_j)}\right]. \tag{PF2}$$

Now split each summand into “the part where the trailing product is replaced by 1” plus “the correction for the trailing product not actually being 1”: $\left(\frac{\pi(y_t|s_t)}{\mu(y_t|s_t)}-1\right)\prod_{j=t+1}^T(\cdot) = \left(\frac{\pi(y_t|s_t)}{\mu(y_t|s_t)}-1\right) - \left(\frac{\pi(y_t|s_t)}{\mu(y_t|s_t)}-1\right)\left(1-\prod_{j=t+1}^T(\cdot)\right)$ — this is just adding and subtracting the same quantity, always valid. Summing this split over $t$ and pulling the reward $R(y)$ back in produces exactly $L'_\mu(\pi)$ (Equation T2, the “replace trailing product by 1” part) minus $\Delta(\mu,\pi)$ (Equation T3, the correction term), which is Theorem 3.1. This derivation makes clear why $\Delta$ vanishes at $\pi=\mu$: every ratio $\pi(y_j|s_j)/\mu(y_j|s_j)$ becomes exactly 1, so both the leading factor $(\cdot - 1)$ and the trailing-product correction $(1-\prod(\cdot))$ are identically zero. It also clarifies what $\Delta$ physically represents: it is the part of the reward change attributable to later tokens’ distributions having shifted, weighted by how much the current token’s ratio deviates from 1 — exactly the higher-order, cross-token interaction that a naive per-token linearization (just summing up $(r_t-1)$ terms, as $L'_\mu$ does) cannot see.

Theorem 3.2 — Two Policy Improvement Bounds

$\Delta(\mu,\pi)$ in Equation T3 is exact but intractable to bound tightly in general (it depends on a product of $T-t$ ratios). The paper proves two different, useful bounds on it, each obtained by a different tightening strategy:

$$J(\pi)-J(\mu) \;\ge\; L'_\mu(\pi) - 2\xi T(T-1)\cdot D_{TV}^{\max}(\mu\|\pi)^2, \tag{T4}$$ $$J(\pi)-J(\mu) \;\ge\; L'_\mu(\pi) - 4\xi\,\bar{D}_{TV}(\mu,\pi), \tag{T5}$$

where $\xi=\max_y|R(y)|$ is the maximum absolute terminal reward, $D_{TV}^{\max}(\mu\|\pi)=\max_{s_t}D_{TV}\big(\mu(\cdot|s_t)\|\pi(\cdot|s_t)\big)$ is the worst-case per-state divergence over the whole sequence, and $\bar{D}_{TV}(\mu,\pi)=\mathbb{E}_{y\sim\mu}\left[\sum_{t=1}^{|y|}D_{TV}\big(\mu(\cdot|s_t)\|\pi(\cdot|s_t)\big)\right]$ is the average, summed per-token divergence actually realized along the sampled trajectory.

flowchart LR
    I0["Exact identity: J(pi) - J(mu) = L'_mu(pi) - Delta(mu,pi)"] --> I1["Delta bounded via worst-case per-state divergence"]
    I0 --> I2["Delta bounded via average per-token divergence"]
    I1 --> B1["Bound T4: penalty grows as T squared times worst-case TV squared"]
    I2 --> B2["Bound T5: penalty grows linearly in average realized TV"]
    B1 -.->|"looser, but structurally mirrors classical TRPO bound"| C["Both justify constrained problem: maximize L'_mu(pi) subject to divergence less than delta"]
    B2 -.->|"tighter for long sequences, matches per-token control used by PPO and DPPO"| C

Figure 3 (math visualization of the derivation chain): the same exact identity is bounded two different ways depending on whether you take a worst-case (max) or average-case (mean) view of per-token divergence. The average-case bound (T5) is what actually motivates DPPO’s per-token masking design, because it penalizes the sum of realized per-token divergences directly — exactly what a per-token mask can control.

Why two bounds instead of one? The max-divergence bound (T4) is structurally analogous to the classical TRPO bound (T3 in the paper’s numbering matches Equation P3 above), with the sequence length $T$ playing the role the effective horizon $\frac{1}{1-\gamma}$ played in the discounted setting — this is reassuring because it shows the LLM-specific derivation reduces to something familiar in form. But it scales as $T(T-1)$, which is punishing for long chain-of-thought responses (thousands of tokens): a single large per-state divergence anywhere in a long sequence blows up the penalty quadratically in length. The average-divergence bound (T5) is tighter for long responses precisely because it only accumulates the divergence that was actually realized, token by token, rather than assuming the worst state achieves the worst-case divergence everywhere. Practically, (T5) is also the bound whose structure matches what a per-token mask (which either passes or blocks each token’s gradient) can directly enforce — this is the theoretical justification for why DPPO’s algorithm, introduced next, operates token-by-token rather than trying to estimate a single sequence-level divergence number.

A numeric sense of scale. Consider a chain-of-thought response of length $T=1000$ tokens (not unusual for a reasoning-heavy math problem), a maximum reward magnitude $\xi=1$ (typical for a normalized 0/1 or bounded reward), and suppose the worst-case per-state TV divergence is a modest $D_{TV}^{\max}=0.01$. The max-divergence bound’s penalty term is $2\xi T(T-1)\cdot(D_{TV}^{\max})^2 = 2\cdot1\cdot1000\cdot999\cdot(0.01)^2 \approx 199.8$ — a penalty roughly two orders of magnitude larger than a typical reward range of $[0,1]$, which would make the bound essentially vacuous (it would allow no meaningful conclusion about improvement) despite the worst single state only having a fairly small $0.01$ TV divergence. Now compare the average-divergence bound: if the average, summed per-token divergence realized along this same trajectory is, say, $\bar{D}_{TV}=2$ (i.e., an average per-token divergence of $0.002$ accumulated additively across 1000 tokens — a plausible, still-small per-token number), the penalty is $4\xi\bar{D}_{TV} = 4\cdot1\cdot2=8$, still non-trivial relative to a $[0,1]$ reward range but dramatically tighter than 199.8. This is exactly the numeric intuition behind the paper’s claim that (T5) is “tighter for long LLM responses”: the max-divergence bound assumes the single worst per-state divergence could in principle occur at every one of the $T(T-1)/2$ ordered pairs of positions, which is enormously pessimistic for a 1000-token sequence, while the average-divergence bound only charges for divergence that was actually realized, once, summed linearly rather than assumed to recur combinatorially.

Both bounds justify the same constrained-optimization template as classical TRPO (Equation P4), just re-derived for this regime:

$$\max_\pi\ L'_\mu(\pi)\quad\text{s.t.}\quad D_{TV}^{\max}(\mu\|\pi)\le\delta \ \ (\text{or an average-divergence variant}). \tag{T6}$$

Divergence Proximal Policy Optimization (DPPO)

One Full Training Iteration, PPO vs. DPPO, Step by Step

Before diving into the formal objective, it helps to walk through exactly what changes, and what stays the same, across one iteration of an RL post-training loop when you swap PPO’s ratio-based clip for DPPO’s divergence-based mask. I lay this out as a concrete sequence of steps, annotating at each point whether PPO and DPPO agree or diverge:

1. Sample a batch of prompts. Identical in both: draw a batch of prompts from the training dataset. Nothing about the trust-region mechanism affects this step.
2. Roll out responses with the inference engine. Identical in both: for each prompt, the inference/rollout engine (using parameters $\theta'$) samples one or more responses, logging the token-level probabilities $\mu_{\theta'}(y_t|s_t)$ it assigned to every token it actually sampled. This logging step is required by both PPO and DPPO — no new instrumentation needed for DPPO here.
3. Score responses and compute the advantage. Identical in both: apply the reward function (rule-based verifier, or a reward model) to each response, then compute $\hat{A}_t$ via whatever advantage estimator is in use (GRPO-style group-relative, as in Equation P6, in all of this paper’s main experiments). The choice of advantage estimator is completely orthogonal to the PPO-vs-DPPO choice.
4. Recompute token probabilities with the training engine. Identical in both: pass the same sampled tokens through the training engine at its current parameters $\theta$, obtaining $\pi_\theta(y_t|s_t)$ for each token. This is also where training-inference mismatch enters, since $\pi_\theta$ and $\mu_{\theta'}$ can differ even at $\theta=\theta'$ due to engine/numerics differences.
5. Compute the per-token ratio $r_t=\pi_\theta(y_t|s_t)/\mu_{\theta'}(y_t|s_t)$. Identical in both — every PPO-family method needs this ratio for the actual gradient term $r_t\hat{A}_t$ regardless of how masking is decided.
6. This is where PPO and DPPO diverge. PPO checks only whether $r_t$ itself falls outside $[1-\epsilon_{low}, 1+\epsilon_{high}]$ and masks based on that single scalar. DPPO additionally computes a divergence estimate $D_t$ — either the Binary form (Equations D3–D4, using the same two scalars $\mu_{\theta'}(y_t|s_t)$ and $\pi_\theta(y_t|s_t)$ already computed in steps 2 and 4, at essentially zero extra cost) or the Top-K form (Equations D5–D6, requiring the rollout engine’s top-$K$ token probabilities, a small extra piece of logged information) — and masks based on whether $D_t$ exceeds $\delta$, conditioned on the update direction.
7. Apply the mask and aggregate the loss. Identical structurally in both: multiply the (possibly masked) ratio-weighted advantage by $\nabla_\theta\log\pi_\theta(y_t|s_t)$, sum over tokens and responses, and take a gradient step. The masking decision differs (step 6); the mechanics of applying whatever mask was decided are the same.
8. Repeat for the configured number of minibatch updates on this batch of rollouts, then sample a new batch of prompts and start again. Identical in both, and this is exactly the point in the loop where accumulated training-inference mismatch (from reusing one rollout batch for several gradient steps) compounds if the trust region is not correctly anchored — which is the RQ1/RQ2 finding above.

The practical upshot of walking through it this way: DPPO’s entire footprint on an existing training loop is confined to step 6. Nothing about data collection, reward scoring, advantage estimation, or gradient aggregation changes — which is exactly why the paper can present it as a drop-in replacement rather than a new training system, and why the “Practical Recipe” checklist later in this review is short.

The DPPO Objective and Mask

DPPO keeps PPO’s cheap, first-order masking structure — no second-order optimization, no explicit Lagrangian — but replaces the quantity being thresholded from a single-token ratio to an actual divergence estimate:

$$L_\mu^{DPPO}(\pi) = \mathbb{E}_{y\sim\mu}\left[\sum_{t=1}^{|y|} M_t^{DPPO}\cdot r_t\cdot\hat{A}_t\right], \tag{D1}$$ $$M_t^{DPPO} = \begin{cases} 0, & \text{if } (\hat{A}_t>0 \text{ and } r_t>1 \text{ and } D>\delta)\ \text{ or }\ (\hat{A}_t<0 \text{ and } r_t<1 \text{ and } D>\delta), \\ 1, & \text{otherwise}, \end{cases} \tag{D2}$$

where $D \equiv D\big(\mu(\cdot|s_t)\|\pi(\cdot|s_t)\big)$ is the estimated divergence (TV or KL) at that state and $\delta$ is a threshold hyperparameter. Substituting $D\to|r_t-1|$ in Equation D2 recovers exactly PPO’s original clip-and-mask rule — which is a good sanity check that DPPO is a strict generalization, not a different algorithm wearing PPO’s clothes.

Pseudocode (per-token masking, one training minibatch):

for each response y = (y_1, ..., y_T) sampled from rollout policy mu:
    for t in 1..T:
        r_t      = pi(y_t | s_t) / mu(y_t | s_t)          # importance ratio, still needed for the gradient itself
        D_t      = estimate_divergence(mu(.|s_t), pi(.|s_t))   # Binary or Top-K approximation, see below
        moving_away = (A_hat_t > 0 and r_t > 1) or (A_hat_t < 0 and r_t < 1)
        if moving_away and D_t > delta:
            M_t = 0        # block: this update would push the policy outside the trust region
        else:
            M_t = 1        # pass: either moving toward mu, or divergence still within budget
        contribution_t = M_t * r_t * A_hat_t
    loss = -mean(sum_t contribution_t)                    # negative because we ascend reward, descend loss

Two design properties are worth calling out explicitly, because they are exactly what makes DPPO retain PPO’s good behavior while fixing its bad behavior. First, the mask is directional: it only ever considers blocking an update that is already moving away from the rollout policy in the direction the advantage sign implies (positive advantage + ratio above 1, i.e., “already increasing a good token’s probability past parity,” or the symmetric case for negative advantage). It never blocks an update moving back toward $\mu$ — e.g., $\hat{A}_t>0$ but $r_t<1$ (a good token whose probability is still below the rollout policy’s) is always allowed through, because that move can only shrink divergence, not grow it. This preserves PPO’s core intuition that the clip should be one-sided in effect. Second, and this is the actual innovation, the decision of whether to block is now based on whether the entire distribution has drifted too far ($D>\delta$), not on how extreme one single sampled token’s ratio happens to look. This directly attacks the two failure modes documented above: a rare token’s huge ratio no longer triggers a block by itself if the actual divergence contribution is small, and a dominant token’s modest-looking ratio no longer escapes masking if it is in fact moving a large amount of probability mass.

Worked Example: Running the DPPO Mask on the $a_{low}$/$a_{high}$ Case

It is worth mechanically checking the mask against the two tokens introduced earlier, to see exactly why DPPO produces the opposite masking decision from PPO on both of them. Take $\delta=0.05$ as the divergence threshold (a representative small value; the paper tunes this per experiment) and assume both tokens received a positive advantage $\hat{A}_t>0$ (i.e., the policy is being encouraged to raise their probability further).

Token $a_{low}$: $\mu=10^{-4}$, $\pi=10^{-2}$, ratio $r=100>1$. Using the Binary TV approximation (Equation D3), $D^{Bin}_{TV} = |10^{-4}-10^{-2}| \approx 0.0099 < \delta=0.05$. The “moving away” condition ($\hat{A}_t>0$ and $r_t>1$) is satisfied, but the divergence condition ($D>\delta$) is not — so $M_t=1$: the update passes through unmasked. Compare to PPO: with $\epsilon=0.2$, $r=100$ is far outside $[0.8,1.2]$, so PPO’s clip fires and this update is heavily suppressed. This is exactly the over-penalization the paper’s diagnosis identifies, and DPPO corrects it.

Token $a_{high}$: $\mu=0.99$, $\pi=0.80$, ratio $r\approx0.808 < 1$. Here the “moving away” condition for a positive advantage requires $r_t>1$, which is false ($r<1$) — so regardless of the divergence value, $M_t=1$ under this specific advantage sign. This may look surprising, but it is consistent with the mask’s design: for a positive-advantage token, a ratio below 1 means the update is increasing probability relative to $\mu$ only weakly, or the training policy has for some other reason moved below the rollout probability — either way it is not the “runaway increase” pattern the mask exists to catch. The genuinely dangerous case for $a_{high}$ arises when its advantage is negative (the policy is being told to suppress an already-dominant token): there, $\hat{A}_t<0$ and $r_t=0.808<1$ satisfies “moving away,” and the divergence $D^{Bin}_{TV}=|0.99-0.80|=0.19 \gg \delta=0.05$ triggers $M_t=0$ — DPPO blocks this update, exactly because collapsing a dominant token’s probability by 0.19 in one step is the kind of large, destabilizing shift Theorem 3.2’s bound is designed to prevent. Under PPO with $\epsilon=0.2$, the same ratio $r\approx0.808$ sits right at the edge of $[0.8,1.2]$ and may or may not clip depending on the exact threshold — an unreliable, close-call outcome for what is actually a large, unambiguous divergence event. This side-by-side walkthrough is the concrete mechanism behind the paper’s abstract claim that DPPO “resolves the over- and under-constraining issues inherent in standard PPO”: the same two tokens receive opposite-from-PPO masking decisions once the trust region is measured against real distributional shift instead of a single ratio.

Binary Approximation: The Cheapest Possible Divergence Estimate

Computing an exact per-state divergence requires the full categorical distribution over the entire vocabulary (often 100k+ tokens) for both $\mu$ and $\pi$ at every generated position — memory-prohibitive at scale. The Binary approximation collapses each distribution to a two-outcome Bernoulli variable: “the token that was actually sampled” versus “everything else,” i.e. $p_{\tilde\pi}^t = \big(\tilde\pi(a_t|s_t),\ 1-\tilde\pi(a_t|s_t)\big)$ for $\tilde\pi\in\{\mu,\pi\}$. Plugging this two-outcome distribution into the standard TV and KL formulas gives closed forms that need only the scalar probabilities of the one sampled token under both policies — exactly the same two numbers PPO already computes to form its ratio, so this is essentially free:

$$D_{TV}^{Bin}(t) = \big|\mu(a_t|s_t) - \pi(a_t|s_t)\big|, \tag{D3}$$ $$D_{KL}^{Bin}(t) = \mu(a_t|s_t)\log\frac{\mu(a_t|s_t)}{\pi(a_t|s_t)} + \big(1-\mu(a_t|s_t)\big)\log\frac{1-\mu(a_t|s_t)}{1-\pi(a_t|s_t)}. \tag{D4}$$

Why does collapsing the entire vocabulary down to one token vs. “everything else” still correctly diagnose the two failure modes above? Because Equation D3 is literally the absolute probability-mass difference of the sampled token — exactly the quantity that was mispriced by the ratio: for $a_{low}$ it evaluates to $|10^{-4}-10^{-2}|\approx 0.0099$ (small, correctly not triggering a block), and for $a_{high}$ it evaluates to $|0.99-0.80|=0.19$ (large, correctly eligible for a block). This is precisely the fix motivated in the diagnosis section above, obtained at essentially zero extra compute cost over vanilla PPO.

Top-K Approximation: A Richer But More Expensive Alternative

The Binary approximation only “sees” the one sampled token; it is blind to shifts among other high-probability tokens that were not sampled this time (e.g., if $\mu$ and $\pi$ swap which of two near-tied tokens is favored, Binary would report near-zero divergence if the sampled token’s own probability barely changed, even though the distribution’s shape shifted substantially). Top-K addresses this by explicitly tracking a small representative set $A'_t = \text{TopK}(\mu(\cdot|s_t), K)\cup\{a_t\}$ — the $K$ highest-probability tokens under the rollout policy, plus the sampled token if it isn’t already in that set — and folding every other token into a single aggregated “other” bucket, forming reduced $(K{+}2)$-outcome categorical distributions $p^t_{\tilde\pi}$ over $A''_t = A'_t\cup\{\text{other}\}$:

$$p_{\tilde\pi}^t(a) = \tilde\pi(a|s_t)\ \ \forall a\in A'_t, \qquad p_{\tilde\pi}^t(\text{other}) = 1-\sum_{a\in A'_t}\tilde\pi(a|s_t). \tag{D5}$$

The divergence is then computed over this reduced $(K{+}2)$-outcome distribution exactly as it would be over the full vocabulary:

$$D_{TV}^{TopK}(t) = \frac{1}{2}\sum_{a\in A''_t}\big|p_t^\mu(a) - p_t^\pi(a)\big|,\qquad D_{KL}^{TopK}(t) = \sum_{a\in A''_t} p_t^\mu(a)\log\frac{p_t^\mu(a)}{p_t^\pi(a)}. \tag{D6}$$

The intuition for why Top-K “better captures changes in the head of the distribution” (as the paper puts it): the terms that dominate a true divergence calculation are almost always the highest-probability tokens (they carry the most mass to move), so explicitly tracking them and lumping only the long, thin tail into one bucket recovers most of the fidelity of the exact computation while keeping the cost at $O(K)$ per token instead of $O(|\text{vocab}|)$. The empirical ablation later in the paper (Section 7.3) shows Binary and Top-K performing similarly — which is itself an informative negative result: it says the head-vs-tail structure that Top-K adds barely matters in practice, and the sampled-token-vs-rest structure that Binary alone captures is already doing almost all of the useful work. For a production system, this is good news: you get most of the benefit at the cheapest possible implementation cost.

Design axis	PPO (baseline)	DPPO-Binary	DPPO-Top-K
Quantity thresholded	single-token ratio $r_t$	2-outcome TV/KL of sampled token vs. rest	$(K{+}2)$-outcome TV/KL over top-$K$ tokens + “other”
Extra compute vs. PPO	— (baseline)	negligible (reuses ratio’s two scalars)	$O(K)$ extra lookups per token, still cheap
Sensitive to non-sampled high-prob. tokens shifting?	no (ratio only sees sampled token)	no	yes
Direction-aware masking	yes (one-sided clip)	yes (one-sided mask)	yes (one-sided mask)
Anchor distribution	recomputed-vs-rollout ambiguous in practice	rollout $\mu_{\theta'}$ (Section 5.2 shows this is required)	rollout $\mu_{\theta'}$

Figure 4 (design comparison, table-as-figure): DPPO’s two approximations both preserve PPO’s cheap, one-sided masking structure while replacing the mispriced ratio with an actual divergence estimate; Top-K adds sensitivity to head-of-distribution reshuffling at a small added cost, but the ablation shows this refinement contributes little beyond what Binary already achieves.

flowchart TB
    R["Rollout engine samples y ~ mu, logs mu(a_t|s_t) for sampled tokens"] --> T["Training engine recomputes pi(a_t|s_t) for the same tokens"]
    T --> C{"Choose approximation"}
    C -->|"Binary (cheap)"| BN["Collapse vocab to {a_t, rest}: compute D from 2 scalars mu(a_t), pi(a_t)"]
    C -->|"Top-K (richer)"| TK["Fetch top-K tokens under mu, form (K+2)-outcome dist for mu and pi, compute D"]
    BN --> M["Mask M_t: block only if D above delta AND update moving away from mu"]
    TK --> M
    M --> G["Masked gradient M_t * r_t * A_hat_t flows into policy update"]

Figure 5 (data-flow / pipeline diagram): the divergence-estimation step is a small, cheap detour inserted between “compute the ratio” (which every PPO-family implementation already does) and “apply the mask” — it does not change the surrounding training loop’s structure, which is why DPPO is a drop-in replacement rather than a new training system.

Why the Approximations Are Principled Lower Bounds, Not Just Heuristics

A natural worry about both Binary and Top-K is that “collapsing the vocabulary” sounds like it could go either way — maybe it underestimates divergence (giving a false sense of safety), maybe it overestimates it (masking updates unnecessarily). The paper’s Appendix B settles this with a short, clean argument, and it is worth reproducing because it upgrades both approximations from “reasonable-sounding heuristics” to “provably conservative estimates.” Let $\mathcal{C}=\{C_1,\dots,C_m\}$ be any partition of the vocabulary into disjoint groups — Binary is the specific partition $\{\{a_t\},\ \mathcal{A}\setminus\{a_t\}\}$ (two groups: the sampled token, everything else), and Top-K is the partition $\{\{a\}:a\in A'_t\}\cup\{\mathcal{A}\setminus A'_t\}$ ($K{+}1$ singleton groups for the top tokens, plus one group for the tail). Define the divergence computed on the partitioned space as $D_{TV}^{\mathcal{C}} = \frac{1}{2}\sum_{j=1}^m |\mu(C_j|s_t)-\pi(C_j|s_t)|$, i.e., exactly what Binary/Top-K compute. The claim is $D_{TV}^{\mathcal{C}}\le D_{TV}(\mu\|\pi)$ for any partition — the approximation can never report more divergence than actually exists, only equal or less. The proof is a direct application of the triangle inequality: for any group $C_j$, $|\mu(C_j|s_t)-\pi(C_j|s_t)| = \left|\sum_{a\in C_j}\big(\mu(a|s_t)-\pi(a|s_t)\big)\right| \le \sum_{a\in C_j}\big|\mu(a|s_t)-\pi(a|s_t)\big|$ — the absolute value of a sum is at most the sum of absolute values, because same-signed terms add constructively while opposite-signed terms partially cancel inside the sum on the left before the outer absolute value is taken, whereas the right-hand side pre-takes every absolute value first and so never benefits from that cancellation. Summing this inequality over all $m$ groups and dividing by 2 gives $D_{TV}^{\mathcal{C}} \le D_{TV}(\mu\|\pi)$ directly. This has a very concrete practical meaning: the gap between the true divergence and the Binary/Top-K estimate is exactly the amount of within-group cancellation the partition throws away — probability mass shifts within the same bucket (e.g., two non-sampled, non-top-$K$ tail tokens swapping probability with each other) are invisible to the approximation because they cancel inside the “other” bucket’s sum before the absolute value is applied, while shifts between buckets (in particular, any shift involving the sampled token itself, which always gets its own singleton bucket in both schemes) are captured exactly. Because the estimate is a lower bound, thresholding $D^{\mathcal{C}}>\delta$ can only under-trigger masking relative to the true divergence, never over-trigger it — which is the direction of error you would choose deliberately if forced to pick one, since it means DPPO’s mask is conservative about blocking gradient (erring toward PPO’s original “when in doubt, let training proceed” bias) rather than being spuriously trigger-happy.

Baselines Explained: PG-IS, PG-TIS/CISPO, MiniRL/MiniRL-TIS

The experiments below compare DPPO against several baselines whose names are easy to skim past without registering what each actually does differently; since the whole point of the experiments is to isolate which design choice causes collapse, it is worth being precise about each one. PG-IS (“policy gradient with importance sampling”) is the simplest possible off-policy correction: multiply the policy-gradient advantage term by the raw per-token ratio $r_t$, with no clip, no mask, no divergence constraint at all — it is what you get if you take Equation P5 and delete the $\min(\cdot,\cdot)$ entirely. It is included specifically as the “no trust region” control condition. PG-TIS, also known in the literature as CISPO (Chen et al., 2025), adds Truncated Importance Sampling: the ratio is capped (clamped to an upper value) but never zeroed out — gradient signal always flows, just at a bounded weight, which is different from PPO/DPPO-style masking where the update can be entirely zeroed. MiniRL and MiniRL-TIS (Zheng et al., 2025) are PPO-style algorithms that do use a clip/mask, but — this is the detail RQ2 is built around — anchor the divergence measurement to a recomputed on-policy distribution $\pi_{\theta'}$ (re-running the just-updated parameters through the training engine) rather than the original rollout distribution $\mu_{\theta'}$ that generated the data. Seeing all four baselines side by side makes the experimental design legible: PG-IS/PG-TIS test “what if there is no real constraint,” and MiniRL/MiniRL-TIS test “what if there is a constraint but it is anchored to the wrong reference distribution” — between them they isolate both of the paper’s central claims (a real trust region is necessary; it must be anchored to $\mu_{\theta'}$) before DPPO is even introduced as a comparison point.

A Unified View: Every Baseline Is One Equation With Different Knobs

Appendix C.1 of the paper makes the comparison between baselines and DPPO much sharper by writing every algorithm’s gradient as one shared template:

$$\nabla_\theta L(\theta) = \mathbb{E}_{y\sim\mu_{\theta'}}\left[\sum_{t=1}^{|y|} M_t \cdot \min\left(\frac{\pi_\theta(y_t|s_t)}{\mu_{\theta'}(y_t|s_t)},\ C\right)\cdot \hat{A}_t \cdot \nabla_\theta \log\pi_\theta(y_t|s_t)\right]. \tag{U1}$$

Every method in the stability experiments is exactly this equation with a different choice of the mask $M_t$ and the truncation cap $C$ — nothing else changes, which makes it very easy to see precisely which single design decision separates a collapsing method from a stable one:

Method	Mask $M_t$	Truncation cap $C$	What changed vs. DPPO
PG-IS	always $1$ (no mask)	$\infty$ (no cap)	no trust region at all
PG-TIS (CISPO)	always $1$ (no mask)	$3$ (ratio truncated)	caps the ratio’s magnitude but never blocks the gradient
GRPO (Clip-Higher)	ratio-based clip using rollout ratio $r_t$, $\epsilon_{high}=0.28$, $\epsilon_{low}=0.2$	$\infty$	masks on the noisy ratio, not on divergence
MiniRL	same clip shape as GRPO, but uses recomputed ratio $r’t=\pi{\theta’}(y_t	s_t)/\pi_\theta(y_t	s_t)$
MiniRL-TIS	same as MiniRL	$3$	wrong anchor and truncation combined
DPPO (ours)	divergence-based mask using $D_t=D(\mu_{\theta’}(\cdot	s_t)|\pi_\theta(\cdot	s_t))$,$\delta=0.15$(TV) or$\delta=0.05$ (KL)

Framed this way, the paper’s ablation ladder becomes very legible: PG-IS → PG-TIS changes only $C$ (still collapses, in fact worse); PG-IS → GRPO changes only $M_t$ from “none” to “ratio-based” (fixes collapse, in the RQ1 experiment); GRPO → MiniRL changes only which distribution the same-shaped mask is anchored to (breaks stability again, in the RQ2 experiment); GRPO/MiniRL → DPPO changes only what quantity the mask thresholds (ratio vs. divergence). Each step in this ladder isolates exactly one variable, which is what makes the paper’s causal claims about why particular designs succeed or fail more convincing than a single end-to-end “our method is better” comparison would be.

Experiments: Dissecting Stability and Efficiency

The paper does not just claim DPPO works better — it runs a genuinely diagnostic sequence of experiments designed to isolate why trust regions matter and which updates cause collapse, before scaling up to the headline comparison. I walk through each in the order the paper poses the underlying research questions.

RQ1: Is a Trust Region Even Necessary at LLM-Scale Learning Rates?

It is tempting to assume that with the tiny learning rates typical of LLM fine-tuning ($\sim10^{-6}$), a trust region is a belt-and-suspenders safeguard that rarely matters in practice. The experiment (fine-tuning DeepSeek-R1-Distill-Qwen-1.5B on 1,460 curated, all-solvable MATH problems, where a perfectly stable algorithm should converge to ~100% training accuracy) says otherwise:

Method	Trust region?	Training-inference mismatch trend	Outcome
PG-IS	none	grows steadily	eventual collapse
PG-TIS (CISPO)	truncated importance sampling only, no principled constraint	grows, worse than PG-IS	collapse, underperforms PG-IS
MiniRL / MiniRL-TIS	trust region, but mis-anchored (see RQ2)	grows despite constraint	collapse
DPPO-Binary-TV / -KL	trust region, correctly anchored	stays low and stable	near-perfect reward, no collapse

Figure 6 (paper Fig. 3, reproduced): reward curves and training-inference mismatch over ~3,500 steps. Unconstrained methods (PG-IS, CISPO) show monotonically increasing mismatch culminating in collapse; DPPO variants hold mismatch near zero throughout and reach near-perfect training reward.

Takeaway 1 (paper’s own framing): a trust region remains essential even at very low learning rates — the accumulation of training-inference mismatch over thousands of steps, not the per-step learning-rate magnitude, is what eventually triggers collapse if left unconstrained.

RQ2: Which Distribution Should the Trust Region Be Anchored To?

A subtlety easy to get wrong in implementation: should $\mu$ in $D(\mu\|\pi)$ be the original rollout distribution $\mu_{\theta'}$ (the one that actually generated the data, using the inference engine’s numerics), or a recomputed distribution $\pi_{\theta'}$ (re-running the same parameters $\theta'$ through the training engine, to strip out inference-engine-specific numerical quirks before comparing to the updated $\pi_\theta$)? Several open-source implementations (and the MiniRL baseline specifically) use the recomputed anchor, reasoning that it removes engine-specific noise from the comparison. The paper shows this reasoning is backwards: MiniRL, despite having a nominal trust region, still collapses in Figure 3/6 above — and a controlled experiment that takes the otherwise-stable DPPO-KL and only changes its anchor from $\mu_{\theta'}$ to the recomputed $\pi_{\theta'}$ reproduces the same collapse (paper’s Figure 4). The reasoning: the true quantity that must stay bounded is how far the currently deployed, data-generating policy has drifted — substituting a recomputed reference silently discards exactly the information (inference-training numerical divergence) that the trust region exists to police, so a recomputed anchor can report “no divergence” even while the actually-deployed rollout policy has drifted substantially. As a bonus, anchoring to the rollout distribution (which was already computed during sampling) means you never need the extra recomputation pass at all — the paper reports this cuts training compute by roughly 25%.

Takeaway 2: the trust region must be anchored to the original behavior/rollout policy $\mu_{\theta'}$; anchoring to any recomputed on-policy distribution reintroduces instability and costs more compute for no benefit.

RQ3: What Specific Updates Actually Cause Collapse?

This is the most operationally useful diagnostic in the paper. Starting from the unstable, unmasked PG-IS algorithm, the authors search for the minimal mask that restores stability, in order to isolate exactly which updates are dangerous. Reasoning that positively-rewarded updates are typically safe (they reinforce already-successful behavior), they restrict attention to negative-advantage updates and test a single-condition mask:

$$M_t = 0 \ \ \text{if}\ \ \hat{A}_t<0 \ \text{ and }\ \mu_{\theta'}(y_t|s_t) - \pi_\theta(y_t|s_t) \ge \delta. \tag{E1}$$

In words: block a token’s gradient only if it is being penalized (negative advantage) and its probability has already dropped by more than $\delta$ between the rollout and the current training policy. With $\delta=0.5$, this single rule is sufficient to fully stabilize training; a looser threshold ($\delta=0.8$) or the recomputed-anchor variant both still collapse. The paper further shows that the fraction of tokens triggering this “bad update” condition is tiny — under 0.5% of all tokens at any given step — yet that small fraction’s prevalence over time correlates tightly with how erratic the reward curve becomes, which is strong evidence for a causal (not merely correlational) link between these specific updates and collapse.

Takeaway 3: training collapse is driven by a small subset (often <0.5%) of updates on negatively-rewarded tokens whose probability has already been pushed down substantially — i.e., the model is being told “you were very wrong to say this” about a token it still considers fairly likely, which the paper hypothesizes corrupts the model’s internal calibration. This is a mechanistic explanation, not just a phenomenological one, and it is exactly the failure mode DPPO’s divergence-based mask is built to catch (a large probability shift on a token, regardless of what its raw ratio happens to look like).

flowchart TD
    A["Sampled token y_t has negative advantage (model is told: this was part of a wrong answer)"] --> B["Gradient step pushes down pi(y_t): probability drops sharply, e.g. by more than 0.5"]
    B --> C{"Was this token actually a low-confidence guess, or something the model was fairly sure about?"}
    C -->|"model was fairly confident in y_t"| D["Large, aggressive correction to a token the model 'believed in' -- risks corrupting internal calibration/knowledge"]
    C -->|"model already had low confidence"| E["Smaller relative disruption -- less risky"]
    D --> F["Training-inference mismatch grows further on subsequent steps"]
    F --> G["Reward curve becomes erratic; repeated bad updates compound"]
    G --> H["Eventual catastrophic collapse"]
    E --> I["Training proceeds stably"]

Figure 12 (RQ3 causal-mechanism diagram): the paper’s proposed explanation for why a small fraction (<0.5%) of updates causes disproportionate damage — the dangerous case is not “the model was penalized,” it is specifically “the model was penalized hard for a token it was fairly confident about,” which plausibly corrupts calibrated internal representations in a way that compounds across subsequent training steps.

Truncated Importance Sampling Backfires

A secondary, somewhat surprising finding: Truncated Importance Sampling (TIS), a popular variance-reduction technique, actually worsens stability in these experiments (PG-TIS and MiniRL-TIS collapse earlier and underperform their non-TIS counterparts). The proposed explanation mirrors the paper’s core diagnosis: TIS truncates the largest importance ratios, and the tokens most likely to produce large ratios are exactly the low-probability tokens the paper has already shown are systematically mispriced — so TIS ends up disproportionately down-weighting gradient signal from precisely the tokens that carry useful exploration information, introducing a biased, harmful signal loss rather than a clean variance reduction.

Efficiency: What Happens If You Relax the Trust Region for Rare Tokens?

The stability experiments establish that a trust region is necessary; a separate line of experiments (fine-tuning Qwen3-1.7B-Base with GRPO+Clip-Higher on the DAPO dataset) asks whether PPO’s clip is too tight for low-probability tokens specifically, by relaxing $\epsilon\to\infty$ for tokens with $\mu(y_t|s_t)<\alpha$ and sweeping $\alpha$:

Relaxation threshold $\alpha$	Training efficiency vs. $\alpha=0$ baseline	Typical probability of clipped tokens
$\alpha=0$ (GRPO baseline, no relaxation)	baseline	mostly below 0.15
$\alpha=0.1$	substantially faster reward growth	shifts upward as $\alpha$ increases

Figure 7 (paper Fig. 6, reproduced): relaxing the clip specifically for tokens the rollout policy assigns probability below 0.1 meaningfully speeds up training, and the tokens PPO’s default clip catches are disproportionately low-probability, high-entropy ones — consistent with prior observations that RL learning signal concentrates in high-entropy (“exploration”/“reasoning”) tokens.

A follow-up experiment asks which side of the clip matters more: relaxing only the upper bound ($\epsilon_{high}\to\infty$, an extreme version of the popular “Clip-Higher” trick) preserves entropy but yields little efficiency gain; relaxing only the lower bound ($\epsilon_{low}\to\infty$, “Clip-Lower”) gives much faster initial learning but eventually suffers entropy collapse; relaxing both simultaneously is the only variant that is both fast and stable. This is a mild but real critique of the popular Clip-Higher heuristic in the literature: the paper’s data suggests the lower bound, not the upper bound, is actually the more important one to loosen for low-probability tokens — the opposite of what “Clip-Higher” as a name implies. DPPO sidesteps picking a side entirely, because its mask is derived from realized divergence and direction rather than a hand-set asymmetric $\epsilon$.

Which Tokens Actually Get Clipped? A Qualitative Look

The abstract statistical claim (“low-probability tokens get over-clipped”) becomes much more concrete once you see which specific tokens a real training run flags. The paper’s Appendix D trains Qwen3-4B-Base with GRPO on DAPO and, at training step 50, dumps the 50 most frequently clipped tokens separately for positively-rewarded and negatively-rewarded samples. Two categories dominate both lists: numerical and mathematical tokens (digits like 1, 4, 6; symbols like +, =, \[, subscript markers like _{), and reasoning/discourse connectives (Wait, Next, Since, However, Thus, Let, Instead, If). This is not a random sample of the vocabulary’s tail — these are exactly the tokens that carry the logical structure of a chain-of-thought math solution: the connective that signals a self-correction (“Wait, that’s not right”), the word that opens a case split (“If $x>0$…”), the digit that instantiates an intermediate numeric result. The practical implication is blunt: on positively-rewarded samples, PPO-style clipping is disproportionately blocking the reinforcement of exactly the tokens that make a correct solution readable and well-structured; on negatively-rewarded samples, it is disproportionately blocking the necessary suppression of the same category of tokens when they appear inside an incorrect reasoning path. Either way, the mechanism that is supposed to keep training safe is instead selectively silencing the learning signal on the vocabulary subset that matters most for reasoning quality — a much sharper way to state the paper’s efficiency claim than “low-probability tokens get clipped more often,” because it shows the clipped tokens are not merely rare, they are functionally important.

Broader Evaluation: Five Configurations, Two RLHF Settings, and an Ablation

At scale, the paper evaluates five model/technique configurations (MoE base, MoE base + rollout-router-replay, MoE “thinking” model, dense 8B base, MoE + LoRA) against GRPO-ClipHigher and CISPO baselines, all anchored to the rollout distribution per the RQ2 finding:

Configuration	Baseline behavior	DPPO-Binary behavior
MoE Base (no R3)	frequent instability	stable, faster reward growth
MoE Base + R3 (rollout-router-replay)	stabilized by R3	DPPO without R3 already outperforms R3-stabilized baseline; DPPO with R3 gains further — benefits are largely orthogonal
MoE “Thinking” model	GRPO-ClipHigher exhibits catastrophic collapse in this configuration	stable
Dense 8B Base	stable but slower	consistently faster convergence, higher final AIME24/25 scores
MoE + LoRA	(see paper Appendix F.2)	consistent gains reported

Figure 8 (paper Figs. 8–9, reproduced): across AIME24 and AIME25 online evaluation during training, DPPO-Binary-KL/TV consistently converges faster and to a higher final score than GRPO-ClipHigher and CISPO; notably, DPPO without the expensive R3 stabilization technique still beats the R3-stabilized baseline, and CISPO or GRPO-ClipHigher exhibit outright catastrophic collapse in at least one configuration each (CISPO in MoE Base without R3; GRPO-ClipHigher in MoE Thinking) while DPPO does not collapse in any tested configuration.

Beyond verifiable-reward math tasks, two RLHF-style alignment experiments (Gemma-2-9B-It on UltraFeedback, Qwen3-4B-Instruct-2507 on HH-RLHF, both scored by Skywork-Reward-Llama-3.1-8B) show DPPO-Binary-TV improving learned reward faster and reaching a higher final reward than GRPO in both settings, and the resulting Qwen3 checkpoint reaches 80.90 length-controlled / 79.93 raw win rate on AlpacaEval 2.0 — reported by the authors as a new community-leaderboard SOTA at time of writing. This generalization beyond verifiable, rule-based rewards to noisy, learned-reward-model optimization is an important robustness signal, since RLHF rewards are exactly the regime where reward-model noise could plausibly interact badly with a divergence-based mask (a noisy reward could, in principle, make the mask fire on tokens that are not actually problematic) — the fact that gains transfer suggests this interaction is not a serious practical issue, at least in the two settings tested.

Finally, the ablation between Binary and Top-K ($K=20$) approximations under identical MoE-Base settings shows the two performing similarly and both clearly beating the baselines — supporting the paper’s claim that the cheap Binary approximation already captures most of the practically useful signal, discussed above in the method section.

How DPPO Fits Into the Broader 2025–2026 RL-for-LLM Landscape

It helps to place this paper against the wave of recent critic-free RL-for-LLM algorithms rather than reading it in isolation. GRPO (Shao et al., 2024) established the group-relative advantage baseline (Equation P6) that removed the need for a learned critic, and DAPO (Yu et al., 2025) refined GRPO’s practical recipe with token-level loss aggregation and the original “Clip-Higher” trick this paper directly critiques. VAPO (2026) went the other direction, reintroducing a carefully-regularized value function to reduce variance further, while REINFORCE++ and RLOO (Ahmadian et al., 2024) explored simpler, critic-free baselines with different variance-reduction tricks around the group-mean subtraction in Equation P6. GSPO (2026) proposed sequence-level rather than token-level importance ratios as a different fix for the same training-inference mismatch problem this paper studies. What distinguishes this paper from essentially all of that lineage is the level of the fix: DAPO, GSPO, VAPO, and RLOO all keep the ratio-based clipping mechanism and instead change how the advantage is estimated or at what granularity the ratio is computed (token vs. sequence); this paper instead argues the clipping trigger itself (a ratio) is the wrong quantity to threshold, and proposes replacing it while keeping everything else (GRPO-style advantage estimation, the min-clip structural template) unchanged. That makes DPPO more of an orthogonal, composable fix than a competing recipe — nothing in the DPPO mask (Equation D2) is incompatible with, say, GSPO’s sequence-level ratio or VAPO’s value-augmented advantage; you could in principle swap DPPO’s divergence-based mask into either of those pipelines in place of their existing ratio-based clip. The paper does not test such combinations directly, which is itself worth flagging as an open question: does divergence-based masking compose cleanly with sequence-level importance ratios, or does aggregating divergence at the sequence level require a different approximation than Binary/Top-K (which are inherently per-token)?

Practical Recipe: Adopting DPPO in an Existing PPO/GRPO Pipeline

For a team currently running a GRPO- or PPO-style RL post-training pipeline, the paper’s results suggest a concrete, low-risk migration path rather than a full rewrite:

1. Keep your existing advantage estimator. DPPO does not change how $\hat{A}_t$ is computed — whatever critic-free (GRPO/RLOO-style) or critic-based estimator you already use plugs directly into Equation D1 unchanged.
2. Anchor the trust region to the rollout policy, not a recomputed one — this is the single highest-leverage change, and it is free. If your current implementation recomputes $\pi_{\theta'}$ before comparing to $\pi_\theta$ (the MiniRL-style pattern the paper shows is actively harmful), simply switching the comparison distribution back to the logged rollout probabilities $\mu_{\theta'}$ both removes a training-cost-inducing recomputation pass (≈25% savings per the paper) and, per RQ2, directly fixes a source of instability — independent of whether you adopt DPPO’s masking rule at all.
3. Replace the ratio-threshold mask with the Binary divergence mask (Equation D2 + D3/D4). This requires only the same two scalars ($\mu(a_t|s_t)$, $\pi(a_t|s_t)$) your existing ratio computation already produces — no new data needs to be logged, no new forward pass is required.
4. Tune $\delta$ starting from the paper’s reported values ($\delta\approx0.05$ for KL, $\delta\approx0.15$–$0.2$ for TV) as a starting point, then sweep locally for your model scale and dataset, since the paper’s own sensitivity analysis is only lightly reported in the main text (see the Critical Assessment below).
5. Only reach for Top-K if you have a specific reason to suspect head-of-distribution reshuffling matters for your task (e.g., very repetitive, templated generation where the same handful of tokens compete for probability mass turn after turn) — the ablation evidence suggests Binary alone captures most of the benefit at a fraction of the engineering cost.
6. Do not pair this with naive Truncated Importance Sampling unless you have separately verified TIS is not degrading your specific setup — the paper’s finding that TIS actively worsens stability by disproportionately down-weighting exactly the low-probability tokens DPPO is designed to protect is a direct, actionable warning against combining the two techniques without further validation.

Frequently Asked Questions

Does DPPO require a critic (value network)? No. The paper’s headline experiments all use GRPO-style, critic-free advantage estimation (Equation P6); the mask in Equation D2 only needs a scalar advantage $\hat{A}_t$ (sign and magnitude), which can come from any advantage estimator, critic-based or not.

Does the divergence estimate need a second forward pass through the model? No additional forward pass is required beyond what PPO/GRPO already does. Computing $\pi(a_t|s_t)$ for the sampled token $a_t$ (needed for the ratio $r_t$ in every PPO-family method) is exactly what both the Binary approximation (Equations D3–D4) and, with one extra lookup for the top-$K$ tokens, the Top-K approximation (Equations D5–D6) need.

How does this interact with an entropy bonus in the loss? The paper does not report combining DPPO with an explicit entropy regularization term, but the mechanism is structurally compatible: an entropy bonus operates on the full output distribution at each step, independent of whether that step’s policy-gradient term is masked by $M_t$. One plausible interaction worth testing (not covered in the paper) is whether DPPO’s improved handling of low-probability, high-entropy “exploration” tokens (per the Appendix D findings above) makes a separate entropy bonus partially redundant, since DPPO already stops suppressing the update signal on exactly those tokens.

Does this apply to multi-turn or agentic RL, where a single episode spans multiple LLM calls and possibly tool-use steps? Partially — the core theory in Theorems 3.1–3.2 is derived for a single finite-horizon token sequence with one terminal reward, which maps most cleanly onto single-turn generation, but the paper does report one concrete multi-turn experiment (Qwen3-1.7B-Base on a multi-turn Sudoku environment from the Gem library; see the Generalization section below) where DPPO’s efficiency advantage over ratio-based PPO holds up. That is encouraging evidence rather than a theoretical guarantee: the underlying identity was not re-derived for a nested, per-turn reward structure, so the multi-turn result should be read as “the empirical benefit transfers to at least one multi-turn setting,” not as “the theory has been extended to cover multi-turn RL.”

What is the actual wall-clock/memory overhead of DPPO versus vanilla PPO/GRPO? The paper describes both approximations as having “negligible overhead” but does not report a direct wall-clock or peak-memory comparison table in the main text; the qualitative argument (Binary needs the same two scalars PPO already computes; Top-K needs one extra top-$K$ lookup) is convincing in principle but a concrete profiling number would make the “negligible” claim independently verifiable rather than asserted.

Generalization Beyond Math Reasoning, and Hyperparameter Sensitivity

Two appendix results round out the empirical picture and deserve more attention than they get in the main text’s Section 7.4–7.5, since they speak directly to how much of DPPO’s benefit is specific to the AIME-style math-RL setting the headline experiments use.

Hyperparameter sensitivity (Appendix F.4). Fine-tuning Qwen3-30B-A3B-Base on DAPO at 8k context, the paper sweeps $\delta\in\{0.10,0.15,0.20\}$ for DPPO-Binary-TV and $\delta\in\{0.05,0.10,0.15\}$ for DPPO-Binary-KL, comparing against GRPO variants swept over their upper clipping bound:

Method	Swept parameter	Sensitivity observed
DPPO-Binary-TV	$\delta\in\{0.10,0.15,0.20\}$	comparable performance across the whole range
DPPO-Binary-KL	$\delta\in\{0.05,0.10,0.15\}$	remains strong across the whole range
GRPO (varying upper clip bound)	$\epsilon_{high}$	noticeably more sensitive to this hyperparameter than DPPO is to $\delta$

Figure 10 (paper Fig. 19, reproduced): DPPO’s divergence threshold $\delta$ is comparatively forgiving to mis-tune within the ranges tested, whereas GRPO’s clipping bound shows more performance variation across its sweep — and every DPPO configuration tested still outperforms every GRPO configuration tested. This partially answers one of the Critical Assessment concerns below (whether $\delta$ is easier or harder to tune than PPO’s $\epsilon$): within the ranges actually swept, it appears easier, though the sweep does not extend to more extreme mis-settings.

Generalization to new model families and task types (Appendix F.5). Beyond the Qwen3 family on verifiable math rewards, the paper tests three additional axes of generalization, each isolating the trust-region mechanism as the only variable changed (both arms use the same GRPO advantage-estimation framework, differing only in ratio-based vs. TV-divergence-based masking):

Generalization axis	Setting	Result
Different model family	OctoThinker-3B-Hybrid-Base on MATH	DPPO-Binary-TV improves efficiency over PPO-Ratio
Abstract reasoning / induction	Qwen3-1.7B-Base on Arc1D and Acre (Gem library)	DPPO-Binary-TV improves efficiency over PPO-Ratio on both tasks
Multi-turn reasoning	Qwen3-1.7B-Base on multi-turn Sudoku-v0-easy (Gem library)	DPPO-Binary-TV improves efficiency over PPO-Ratio

Figure 11 (paper Fig. 20, reproduced): across four qualitatively different task types — a different base model family, abstract symbolic-reasoning tasks, an induction task, and a genuinely multi-turn environment — swapping only the trust-region masking mechanism from ratio-based to TV-divergence-based (holding the rest of the GRPO recipe fixed) improves training efficiency and, in some settings, final asymptotic performance. This is the strongest evidence in the paper that the diagnosis (ratio clipping is a bad divergence proxy) is a general property of LLM RL rather than an artifact specific to AIME-style math reasoning.

The consistent experimental design here — same base algorithm, single-variable swap — is methodologically the right way to support a generality claim, and it is worth noting explicitly since the paper’s abstract otherwise reads as heavily math-RL-focused; the abstract-reasoning, induction, and multi-turn results are easy to miss if a reader only skims the headline AIME24/25 figures in Section 7.1.

Limitations and Boundary Conditions the Paper Acknowledges

The paper is reasonably candid about scope, and it is worth enumerating the boundary conditions it states explicitly rather than treating “the paper has limitations” as a vague gesture:

• It positions DPPO’s algorithmic contribution as orthogonal to, and combinable with, engineering-level mitigations for training-inference mismatch (higher-precision inference, careful kernel/implementation alignment across engines) — it does not claim to be a substitute for those efforts, only a robust algorithmic layer that still helps even when some numerical mismatch is unavoidable.
• This matters specifically for low-precision, high-throughput inference settings, where perfect numerical alignment between the rollout and training engines is not an option for cost reasons — the paper frames DPPO as most valuable precisely in that regime, rather than as a reason to stop investing in engine-alignment engineering.
• It explicitly frames the Binary and Top-K divergence estimates as lower bounds on the true divergence (proven in the paper’s Appendix B), meaning DPPO’s threshold $\delta$ is calibrated against an underestimate of the real distributional shift.
• This is a deliberate, stated design choice rather than an oversight, but it does mean the effective trust region enforced in practice is somewhat looser than the nominal $\delta$ value suggests — a reader tuning $\delta$ by analogy to a true-divergence threshold from another source would be systematically miscalibrated.
• The paper’s Impact Statement is minimal (a boilerplate one-line acknowledgment of “many potential societal consequences… none which we feel must be specifically highlighted”), which is standard for a methods paper of this kind but means there is no explicit discussion of, for instance, whether more efficient and stable RL training could accelerate deployment of capabilities whose societal effects are still being worked out — a point that applies to essentially all RL-efficiency papers in this space, not a specific failing of this one.

Critical Assessment: Weaknesses & Improvements

Weaknesses and unconvincing elements.

• The headline theoretical contribution (Theorems 3.1–3.2) establishes a sufficient condition for improvement via a lower bound, but the paper never empirically measures how tight that bound is in a real training run.
• We see that DPPO’s masking heuristic, which is derived from the bound’s structure, works well in practice — but we never see a plot of the actual gap between $J(\pi)-J(\mu)$ and the bound’s right-hand side over the course of training, which is the concrete evidence that would tell us whether the theory is doing real predictive work or is mainly a motivating narrative for an otherwise reasonable heuristic.
• The threshold $\delta$ (the divergence budget) and $K$ (for Top-K) are treated almost as free hyperparameters in the main text; Section 7.5 promises a hyperparameter sensitivity study, but it is relegated entirely to an appendix not analyzed in the main paper.
• This is a notable gap given that the whole point of the paper is that PPO’s single hyperparameter ($\epsilon$) is badly miscalibrated for LLMs — the reader is left unable to judge from the main text alone whether DPPO’s own hyperparameter ($\delta$) is easier or harder to tune well across new models and datasets (my own read of the appendix sweep, in the Generalization section above, suggests it is more forgiving, but that is a conclusion I had to reconstruct, not one the paper foregrounds).
• The RQ3 “bad update” diagnostic (Equation E1) is demonstrated using one specific small model (DeepSeek-R1-Distill-Qwen-1.5B) on one specific dataset (1,460 curated MATH problems chosen so every problem is solvable by the initial model) — an unusually clean, low-noise setting engineered to make the diagnosis crisp.
• It is plausible, but not directly shown, that the same $<0.5\%$-of-updates finding generalizes to noisier reward settings (e.g., partially-solvable problem sets, or RLHF reward models with genuine label noise), where “the policy dropped a token’s probability a lot” might be a much less reliable signal of an actually bad update.
• Rollout router replay (R3) — a technique that periodically re-synchronizes MoE expert-routing decisions between the rollout and training engines specifically to prevent routing-induced training-inference mismatch in Mixture-of-Experts models — is treated as an external, orthogonal technique the paper “benefits from” (Section 7.1), but the paper never analyzes why DPPO without R3 already outperforms R3-stabilized GRPO. Is DPPO simply more robust to routing-mismatch noise in general, or does it happen to sidestep specifically the failure mode R3 was designed for? The paper reports the empirical fact but does not diagnose the mechanism, which is a missed opportunity given how central MoE routing instability has become in 2025–2026 LLM RL practice.

Limitations the paper understates.

• The Binary approximation’s blind spot — it cannot see divergence among non-sampled high-probability tokens reshuffling relative to each other — is acknowledged only implicitly, via the existence of the richer Top-K variant, rather than analyzed directly with a constructed counterexample.
• Given that Binary and Top-K perform similarly in the one ablation reported, the paper implicitly concludes this blind spot rarely matters in practice, but a paper making a claim this central to its cost/fidelity trade-off would be stronger with an explicit adversarial or synthetic example showing when Binary’s blindness to head-reshuffling does bite, so practitioners know when to reach for Top-K instead.
• All of the large-scale efficiency and stability comparisons are run against GRPO-ClipHigher and CISPO specifically; several other trust-region variants from 2025–2026 (mentioned only in the related-work discussion, e.g. adaptive KL-clipping methods in the spirit of Wang et al.) are not included as empirical baselines at the same experimental scale.
• So “DPPO beats the current standard heuristics” is well-supported by the data presented, while the stronger claim “DPPO beats every reasonable divergence-aware alternative” is not directly tested against the full frontier of 2025–2026 methods.

Concrete improvement suggestions.

1. Report the empirical tightness of the policy-improvement bound (Theorem 3.2) directly — e.g., track $J(\pi)-J(\mu)$ against $L'_\mu(\pi)$ minus the divergence penalty across a training run — to substantiate that the theory is load-bearing rather than motivational.
2. Move the hyperparameter sensitivity analysis for $\delta$ and $K$ into the main paper with a clear recommended default and a discussion of how sensitive final performance is to it, ideally across at least two model scales, since this is the exact kind of “new heuristic hyperparameter replacing an old one” concern a skeptical reader will raise.
3. Construct one deliberately adversarial synthetic example where Binary’s sampled-token-only view fails to catch a real divergence spike (e.g., two near-equal-probability tokens swapping rank without the sampled token’s own probability changing much) to give practitioners a concrete decision rule for when Top-K is worth its extra cost.
4. Extend the RQ3 “which updates cause collapse” diagnostic to a noisier reward setting (partial-credit rewards, or a learned reward model) to test whether the $<0.5\%$ finding, and the specific threshold that worked there, transfers, since that experiment is currently a single clean case study standing in for a general claim.
5. Include at least one other recent divergence- or entropy-aware baseline from the same time window at the full large-scale experimental setup (not just cited in related work) so the “outperforms existing methods” claim in the abstract and conclusion is measured against the current frontier, not only against GRPO-ClipHigher and CISPO.
6. Directly diagnose why DPPO without R3 outperforms R3-stabilized GRPO — is it a general robustness-to-routing-noise property of divergence-based masking, or an incidental side effect? An ablation isolating routing-related mismatch from other sources of training-inference mismatch would answer this.

A Note on Venue and Timing

This paper appears at ICML 2026 (PMLR volume 306), a peer-reviewed venue, which means the theoretical proofs (Appendix A, not fully reproduced in this review beyond the Theorem 3.1 telescoping sketch) and the experimental protocol have been through external review — worth keeping in mind when weighing how much scrutiny to apply versus a same-topic arXiv preprint that has not yet been reviewed. The arXiv identifier (2602.04879, v3) indicates at least two revision rounds prior to the version reviewed here, which is consistent with a paper that has already absorbed at least one round of external feedback before this review’s own critical assessment.

Follow-Up Research Directions This Work Opens Up

Beyond the specific improvement suggestions in the Critical Assessment above, the paper’s diagnosis and fix suggest several broader research directions that are not criticisms of the paper itself, but natural next steps it sets up:

• Extending the finite-horizon theory to multi-turn and hierarchical reward structures. As discussed in the FAQ, the core theorems assume one terminal reward per sequence; a formal treatment of nested horizons (per-turn rewards inside a multi-turn episode, or per-tool-call rewards inside an agentic trajectory) would put the promising empirical Sudoku result on the same rigorous footing as the single-turn math results.
• Adaptive or learned divergence thresholds. $\delta$ is currently a fixed hyperparameter per divergence type; given how central the threshold is to the mask’s behavior, a natural next step is making $\delta$ adaptive — e.g., scheduled over training, or set per-layer/per-token-position based on some cheaply estimated local statistic — rather than a single global constant.
• Combining DPPO’s divergence-based masking with sequence-level importance ratios (as in GSPO). As noted in the “How DPPO Fits Into the Broader Landscape” section, these two lines of work operate at different granularities (per-token divergence vs. sequence-level ratio) and are not obviously incompatible; a combined method could in principle inherit benefits from both.
• A theoretically grounded middle ground between Binary and Top-K. Given the Appendix B lower-bound proof generalizes to any vocabulary partition, there is a whole design space of partitions between the two extremes tested (2-outcome Binary, $(K{+}2)$-outcome Top-K) — e.g., partitioning by token frequency buckets, or by semantic/syntactic token class — that the paper’s own theory licenses but does not explore.
• Understanding the interaction between DPPO and explicit entropy regularization (raised in the FAQ above) empirically, given the mechanistic link this paper draws between low-probability, high-entropy tokens and useful exploration signal.
• A mechanistic account of why divergence-based masking is more robust to MoE routing mismatch than ratio-based masking with R3, since the paper reports the empirical fact (DPPO without R3 beats R3-stabilized GRPO) without a causal explanation.

Reproducibility Notes

The paper reports concrete, checkable settings: base models are all publicly available (Qwen3 family at 1.7B/8B/30B-A3B scales, DeepSeek-R1-Distill-Qwen-1.5B, Gemma-2-9B-It, Qwen3-4B-Instruct-2507, plus a Llama-family generalization check in the appendix); datasets are public (MATH, DAPO-Math, UltraFeedback, HH-RLHF); the reward model for RLHF experiments is the public Skywork-Reward-Llama-3.1-8B; and the authors link a GitHub repository (sail-sg/Stable-RL) for the implementation. The core algorithmic change (Equations D1–D2 plus either D3–D4 or D5–D6) is small enough to retrofit into an existing PPO/GRPO training loop without a new training system — the main engineering requirement is exposing per-token probabilities from both the rollout/inference engine and the training engine for the same sampled tokens (which most modern RL-for-LLM stacks already log for the ratio computation) and, for Top-K, also exposing the top-$K$ token probabilities from the rollout engine’s output distribution.

The paper’s Appendix E discloses the exact large-scale training hyperparameters, which are worth recording verbatim since they are the concrete starting point for anyone reproducing this on similar hardware:

Hyperparameter	MoE Base	MoE Base + R3	MoE Thinking	Dense Base	MoE Base + LoRA
Max prompt length	1024	1024	1024	1024	1024
Max response length	16384	16384	16384	8000	8000
Train batch size	256	256	256	128	128
PPO mini-batch size	32	32	32	32	16
Learning rate	$1\times10^{-6}$	$1\times10^{-6}$	$1\times10^{-6}$	$1\times10^{-6}$	$1\times10^{-5}$
Rollout temperature	1.0	1.0	1.0	1.0	1.0
Rollout samples per prompt ($n$)	16	16	16	8	8

Figure 9 (reproduced from paper Table 1): the divergence threshold $\delta$ used for DPPO-Binary-KL is $\delta=0.05$ across all five scaling configurations; for DPPO-Binary-TV it is $\delta=0.2$ for four configurations and $\delta=0.15$ specifically for MoE Base + LoRA. Online evaluation for AIME24/AIME25 uses temperature $0.7$, top-$p$ $0.95$, and $n=32$ samples per question, averaged. Note the learning rate of $1\times10^{-6}$ for four of five configurations directly matches the RQ1 experimental premise (a trust region is still needed even at very low learning rates) — these are not toy learning rates chosen to make the ablation dramatic, they are the actual production-scale settings.

Two practical knobs to expect to tune when reproducing this on a new model/dataset are the divergence threshold $\delta$ (note KL and TV use different natural scales — $0.05$ vs. $0.15$–$0.2$ in the paper’s own settings, so do not reuse a $\delta$ value across the two divergence choices without re-tuning) and, if using Top-K, the choice of $K$ (the paper’s ablation uses $K=20$). The paper reports both divergence choices (Binary-TV vs. Binary-KL) perform comparably in its settings but does not claim this transfers universally to arbitrary new model families or reward structures.

Related Work Map: Who’s Who in This Paper’s Citations

This review cites or discusses a fairly dense web of prior methods; here is a compact map of what each one contributes and where it sits relative to DPPO, useful as a navigational reference if you go on to read the full paper or its appendices.

• TRPO (Schulman et al., 2015) — the original trust-region policy optimization method; proves the classical policy-improvement bound (Equation P3) this paper re-derives for the LLM regime.
• PPO (Schulman et al., 2017) — the clipped first-order approximation to TRPO that this paper’s core diagnosis targets (Equation P5).
• GRPO (Shao et al., 2024) — critic-free, group-relative advantage estimation (Equation P6); treated as the shared advantage-estimation backbone underneath both the baselines and DPPO in this paper.
• DAPO (Yu et al., 2025) — introduced the “Clip-Higher” trick (asymmetric clip bounds) that this paper shows relaxes the wrong side of the clip for low-probability tokens.
• CISPO (Chen et al., 2025) — truncated importance sampling in place of clipping; appears as the PG-TIS baseline, shown here to collapse under the RQ1 stability test.
• MiniRL (Zheng et al., 2025) — a PPO-style method anchored to a recomputed on-policy distribution rather than the rollout distribution; central to the RQ2 “wrong anchor” finding.
• Rollout Router Replay / R3 (Ma et al., 2025) — an MoE-specific stabilization technique that synchronizes expert routing between rollout and training engines; shown to be largely orthogonal to, and combinable with, DPPO’s benefit.
• VeRL (Sheng et al., 2024) — the open-source RL training framework used for all of this paper’s large-scale experiments.
• Skywork-Reward-Llama-3.1-8B (Liu et al., 2024) — the public reward model used for the RLHF-style alignment experiments (UltraFeedback, HH-RLHF).
• Gem library (Liu et al., 2025e) — source of the abstract-reasoning (Arc1D), induction (Acre), and multi-turn (Sudoku) generalization environments used in Appendix F.5.
• Oat framework (Liu et al., 2025c) — the training framework used for the Appendix F.5 generalization experiments outside the main VeRL-based scaling runs.
• Wang et al. (2019; 2020) — earlier work on adaptive KL-based clipping for classical RL, cited as the closest conceptual precedent for constraining divergence directly rather than a ratio, predating the LLM-specific case this paper addresses.
• Achiam et al. (2017), Kakade & Langford (2002), Hunter & Lange (2004) — the foundational theory (Constrained Policy Optimization, the original performance-difference theorem, and the Minorize-Maximization framework) underlying the classical bound this paper re-derives.
• Yao et al. (2025), Qi et al. (2025b), Zheng et al. (2025) — the cluster of work that first characterized and named training-inference mismatch as a distinct failure mode in LLM RL, motivating both TIS-style mitigations and this paper’s own re-derivation of the trust region.
• Yu et al. (2025) — introduced the Clip-Higher trick as part of the DAPO system, directly discussed in the Related Heuristics subsection above as one of the two symptom-level patches this paper supersedes.
• Chen et al. (2025) — proposed CISPO, the truncated-importance-sampling method that appears throughout the stability experiments (as PG-TIS) and is shown to collapse under the RQ1 diagnostic in Section 5.
• Ma et al. (2025) — proposed Rollout Router Replay (R3), the MoE-specific stabilization technique discussed in the Broader Evaluation section and again in the Critical Assessment’s R3-orthogonality weakness point.
• Zheng et al. (2025) — proposed MiniRL, the recomputed-anchor baseline central to the RQ2 finding that anchoring to $\pi_{\theta'}$ rather than $\mu_{\theta'}$ reintroduces instability.
• Sheng et al. (2024) — developed VeRL, the open-source RL post-training framework used to run all of this paper’s large-scale scaling experiments (Section 7, Appendix E).

Paper Section Map

If you go on to read the original paper directly, here is how its section numbering lines up with this review, so you can jump straight to the primary source for any part that interests you most:

Paper section	Content	Covered in this review
1. Introduction	motivation, the $a_{low}/a_{high}$ numeric example	The Core Diagnosis
2. Background	MDPs, Kakade–Langford identity, TRPO	Prerequisites (From MDPs to Policy Gradients through TRPO)
3. Trust Region Under LLM Regime	Theorems 3.1, 3.2	Theory: Trust Regions Rebuilt for the LLM Regime
4. Methodology	PPO recap, DPPO mask, Binary/Top-K approximations	Divergence Proximal Policy Optimization (DPPO)
5. Analysis on Training Stability	RQ1–RQ3, TIS pitfalls	Experiments: RQ1 through Truncated Importance Sampling Backfires
6. Analysis on Training Efficiency	low-probability-token clip relaxation	Efficiency: What Happens If You Relax the Trust Region for Rare Tokens?
7. Broader Evaluation	five configurations, RLHF, ablations, generalization	Broader Evaluation, Generalization Beyond Math Reasoning sections
8. Related Work	Clip-Higher/CISPO connections, training-inference mismatch literature	Related Heuristics subsection, Related Work Map
9. Conclusion	summary	Conclusion (this review)
Appendix A	full proofs of Theorems 3.1–3.2	Proof Sketch subsection (abbreviated derivation, not the full formal proof)
Appendix B	Binary/Top-K as provable lower bounds	Why the Approximations Are Principled Lower Bounds
Appendix C	unified gradient formulation, exact baseline hyperparameters	A Unified View: Every Baseline Is One Equation
Appendix D	qualitative analysis of clipped tokens	Which Tokens Actually Get Clipped?
Appendix E	scaling-experiment training details, hyperparameter table	Reproducibility Notes
Appendix F	extended ablations, hyperparameter sensitivity, generalization experiments	Generalization Beyond Math Reasoning, and Hyperparameter Sensitivity

Before vs. After: How This Paper Changes the Default RL Recipe

A concise before/after comparison of default practice, synthesizing the whole review into one reference table:

Design decision	Common practice before this paper	This paper’s recommended practice
What the trust region measures	single-token probability ratio $r_t$	direct (Binary or Top-K) estimate of policy divergence $D_t$
Which distribution anchors the trust region	often a recomputed on-policy distribution $\pi_{\theta'}$ (e.g., MiniRL-style)	the original rollout distribution $\mu_{\theta'}$
How to handle low-probability tokens	clip aggressively whenever the ratio is large, regardless of actual mass moved	mask only if the actual divergence contribution exceeds $\delta$
How to handle high-probability tokens	often escape clipping if the ratio stays inside $[1-\epsilon,1+\epsilon]$, even with large mass shifts	masked whenever their large mass shift produces $D_t>\delta$, regardless of how “moderate” the ratio looks
Variance-reduction technique for the ratio	Truncated Importance Sampling (TIS), assumed to help stability	avoid TIS unless independently verified — it can worsen stability by suppressing signal from exactly the tokens that need it
Extra recomputation pass for a “clean” reference distribution	common, at added training cost	unnecessary — anchoring to the already-computed rollout distribution is both cheaper and more stable
Root-cause framing of instability	”RL for LLMs is inherently unstable at scale"	"instability concentrates in a small, identifiable, maskable class of updates”

Common Misreadings to Avoid

A few clarifications worth stating explicitly, since a paper this dense in acronyms and equations invites a few predictable misunderstandings:

• DPPO is not a new advantage estimator. It says nothing about how $\hat{A}_t$ is computed; it is entirely about the masking/clipping mechanism applied to an advantage-weighted gradient that some other estimator (GRPO, RLOO, a critic-based method) already produced.
• DPPO is not a critic-based method, and adopting it does not require training a value network. All of the paper’s headline experiments are critic-free (GRPO-style).
• DPPO’s divergence estimate is not the exact TV/KL divergence over the full vocabulary — both Binary and Top-K are provable lower bounds on the true divergence (Appendix B), not the true value itself; the mask is conservative in the specific sense of never over-estimating how far the policy has drifted.
• “DPPO recovers PPO as a special case” is a mathematical statement about substituting $D\to|r_t-1|$ into the mask (Equation D2), not a claim that DPPO and PPO behave identically in the typical case — in practice, on the numeric examples worked through above, DPPO and PPO frequently make opposite masking decisions on the same token.
• The paper’s efficiency claims (relaxing the clip for low-probability tokens) and its stability claims (anchoring to the rollout distribution, masking large negative-advantage shifts) are two separate contributions that happen to be unified by the same algorithm — a team could in principle adopt only the anchoring fix (RQ2) without adopting the full divergence-based mask, and still see some of the stability benefit, though not the efficiency benefit tied to correctly handling low-probability tokens.
• “Sea AI Lab” and “NUS” authorship, and the sail-sg/Stable-RL repository name, refer to the implementation this specific paper ships — DPPO as an algorithmic idea is independent of that particular codebase, and the mask (Equation D2) is simple enough to reimplement directly in most existing RL-for-LLM training frameworks without depending on that repository.

Conclusion

This paper’s contribution is best understood as fixing a bug that has been silently present in essentially every PPO-family LLM RL recipe since PPO was first repurposed for language model fine-tuning: the trust-region safeguard everyone relies on for stability is, at LLM vocabulary scale, actually enforcing a constraint on a wildly noisy single-token proxy rather than on the real quantity it was designed to bound. The fix is conceptually simple — swap the proxy for a cheap, direct estimate of divergence — but getting there required re-deriving the classical policy-improvement theory for a regime (finite-horizon, undiscounted, sequence-terminal-reward) the original theorems were never built for, and then running a careful diagnostic study to confirm the mechanism (a small fraction of large, negative-advantage, probability-collapsing updates) rather than just proposing a plausible-sounding heuristic.

If I had to compress this review into the handful of claims worth remembering after everything else fades, they would be:

1. A trust region for LLM RL should constrain actual policy divergence, not a single sampled token’s probability ratio — the ratio is a legitimate proxy only when the action space is small; at LLM vocabulary scale it becomes a high-variance, systematically biased estimator that fails in two opposite directions at once.
2. Anchor the trust region to the rollout policy that actually generated the data, never to a recomputed on-policy distribution — this single choice is both free (no extra compute) and load-bearing for stability.
3. Training collapse in LLM RL is not diffuse; it concentrates in a small, identifiable class of updates (large, negative-advantage, probability-collapsing), which means it is a tractable problem to detect and mask rather than an inherent, unavoidable cost of scale.
4. The cheapest fix (Binary divergence, reusing scalars you already compute) captures most of the available benefit — sophistication (Top-K, or anything more elaborate) is available if you need it, but is not obviously necessary as a starting point.
5. Heuristic patches that treat only the symptom (Clip-Higher, CISPO-style truncation) are not a substitute for re-examining whether the underlying mechanism they are patching is even measuring the right thing.

For practitioners running RL post-training today, the most actionable near-term takeaways are: anchor your trust region to the rollout policy, not a recomputed one; be suspicious of truncated importance sampling as a stability aid; and consider that the cheapest possible divergence proxy (Binary) may already buy most of the available benefit over a naive ratio clip, at essentially no additional engineering cost. For researchers, the more durable lesson is methodological: when an algorithm inherited from a different regime (small discrete action spaces) gets scaled up by many orders of magnitude (LLM vocabularies) without its core approximation being re-examined, the resulting silent mismatch can persist for years — as it apparently did here — until someone goes back to first principles and asks whether the original theory ever actually applied.