The Mirage of Optimizing Training Policies: Monotonic Inference Policy Improvement for LLM RL

Review date: 2026-07-06 Review author: Zhongzhu Zhou Paper reviewed: The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning Paper authors: Jing Liang, Hongyao Tang, Yi Ma, Yancheng He, Weixun Wang, Xiaoyang Li, Ju Huang, Wenbo Su, Jinyi Liu, Yan Zheng, Jianye Hao, Bo Zheng arXiv: https://arxiv.org/abs/2606.29526 Venue/Status: Preprint (arXiv, June 2026)

Short Answer

This paper exposes a quiet but consequential flaw at the heart of LLM reinforcement learning post-training: virtually every existing method optimizes the training policy $\pi$, yet the policy that actually matters at deployment is the inference policy $\mu$ — and these two policies, even when nominally synchronized, assign systematically different probabilities to the same trajectories due to FP8 quantization, precision discrepancies, and other engine-level differences between production inference stacks (vLLM, SGLang) and training frameworks (FSDP, Megatron). The authors call this gap training-inference mismatch and prove that training-side monotonic improvement does not imply inference-side monotonic improvement. Their proposed fix, MIPU (Monotonic Inference Policy Update), decomposes the inference-side improvement into three auditable terms, addresses each with a concrete algorithmic step — a sampler-referenced importance weight correction and an inference-gap-aware rollback mechanism — and validates the approach on FP8-quantized rollouts of Qwen3-4B and Qwen3-1.7B on math benchmarks, achieving gains of roughly 2–3 percentage points over GRPO while dramatically stabilizing training dynamics.

Prerequisites

Before diving into the paper’s contributions, I want to build the necessary background for readers who are comfortable with machine learning but may not have studied LLM post-training RL in depth. The concepts here are load-bearing for understanding why the paper’s central claim is non-trivial.

LLM Post-Training RL as a Markov Decision Process

Large language model alignment via reinforcement learning reframes the text generation process as a sequential decision problem. At each step, the model (acting as a policy $\pi_\theta$) observes the current context — the input prompt plus all tokens generated so far — and selects the next token to emit. When generation terminates (via an end-of-sequence token or a length limit), a reward function $r$ evaluates the complete output.

More formally, we set up a Markov Decision Process (MDP) as follows:

• State $s_t$: the concatenation of the prompt and all tokens generated through step $t$. This makes the state space exponentially large — a vocabulary of size $V$ and a maximum sequence length of $L$ gives a state space of order $V^L$.
• Action $a_t$: the next token chosen from the vocabulary $\mathcal{V}$.
• Policy $\pi_\theta(a_t \mid s_t)$: the LLM’s conditional distribution over $\mathcal{V}$, parameterized by the model weights $\theta$.
• Reward $R$: a scalar signal received at the terminal state, encoding task quality (e.g., correctness of a math proof verified by symbolic execution, quality of a code solution measured by test pass rate).
• Episode: one complete sequence from prompt to end-of-sequence token.

Because the reward is typically only available at the end of an episode (sparse reward), and because the action space is enormous (vocabulary sizes of 32k–128k are common), LLM RL is a challenging instance of the general RL problem. The MDP structure is also unusual: the state transitions are deterministic given the action (the next state is the previous state with one token appended), and the stochasticity lives entirely in the policy’s token choices.

The key objective is to maximize the expected cumulative reward:

$$J(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\left[ R(\tau) \right] \tag{1}$$

where $\tau = (s_0, a_0, s_1, a_1, \ldots, a_{T-1}, s_T)$ denotes a complete trajectory (token sequence) sampled by unrolling the policy autoregressively from a prompt $q$.

Note that because rewards are sparse (given only at the end), there is no discounting needed in the typical LLM RL formulation — every intermediate token contributes to the final reward equally in the sense that no token is discounted away.

Policy Gradient: The Foundation

Before trust-region methods, the simplest approach to maximizing $J(\pi_\theta)$ is the REINFORCE (Williams 1992) policy gradient estimator:

$$\nabla_\theta J(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\left[ \sum_{t=0}^{T-1} \nabla_\theta \log \pi_\theta(a_t \mid s_t) \cdot G_t \right] \tag{2}$$

where $G_t = \sum_{t'=t}^T r_{t'}$ is the return from step $t$ onward. The key insight is that this gradient is an expectation over rollouts: we generate a trajectory, observe the return, and push the log-probabilities of actions upward (if the return is positive) or downward (if negative).

However, raw REINFORCE has very high variance. Two standard remedies are:

1. Baseline subtraction: replace $G_t$ with $(G_t - b)$ for some state-dependent baseline $b(s_t)$. This yields the advantage $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$, which captures how much better action $a$ is relative to the average in state $s$. Baselines do not introduce bias in expectation but dramatically reduce variance.

2. Actor-critic: train a separate “critic” network to estimate $V^\pi(s)$ or $Q^\pi(s,a)$, providing a lower-variance advantage estimate. PPO uses a critic; GRPO avoids it.

PPO, TRPO, and GRPO

The dominant algorithms for LLM post-training RL derive from the policy gradient family, with trust-region constraints added for stability.

TRPO (Trust Region Policy Optimization) derives from the observation that standard gradient ascent on $J(\pi_\theta)$ can take steps that completely destroy the policy if the learning rate is too large. TRPO constrains each update to a trust region defined by a KL divergence bound:

$$\max_\theta \; \mathbb{E}_{(s,a) \sim d^{\pi_{\text{old}}}}\left[ \frac{\pi_\theta(a \mid s)}{\pi_{\text{old}}(a \mid s)} A^{\pi_{\text{old}}}(s,a) \right] \quad \text{s.t.} \quad D_\text{KL}(\pi_{\text{old}} \| \pi_\theta) \leq \delta \tag{3}$$

This is a constrained optimization problem that requires computing (or approximating) the Fisher information matrix of the policy — an $O(|\theta|^2)$ operation that is infeasible for billion-parameter LLMs.

PPO (Proximal Policy Optimization) approximates TRPO’s constraint through a clipped surrogate objective, converting the constrained problem into an unconstrained one that can be solved with first-order methods (Adam, AdamW):

$$\mathcal{L}^{\text{PPO}}(\theta) = \mathbb{E}_{(s,a) \sim d^{\pi_{\text{old}}}}\!\left[ \min\!\left( r(\theta) \hat{A},\; \text{clip}(r(\theta), 1\!-\!\epsilon, 1\!+\!\epsilon) \hat{A} \right) \right] \tag{4}$$

where $r(\theta) = \pi_\theta(a \mid s) / \pi_{\text{old}}(a \mid s)$ is the probability ratio, $\hat{A}$ is an advantage estimate, and $\epsilon$ is a clipping threshold (typically 0.1–0.2). The clipping prevents ratios from drifting too far from 1: if $r(\theta)$ would be pushed above $1+\epsilon$ by a gradient step, the clipping stops it. This is a pessimistic bound — it never lets the policy exploit advantageous actions beyond the trust region.

The key property of PPO’s surrogate: when the advantage is positive (action was good), clipping prevents unbounded increase of $r(\theta)$; when the advantage is negative (action was bad), clipping prevents $r(\theta)$ from being pushed below $1-\epsilon$ (you do not want to completely suppress actions that only slightly underperformed). This makes the objective a lower bound on the true policy improvement.

GRPO (Group Relative Policy Optimization) is a PPO variant designed specifically for LLM RL, introduced in DeepSeekMath and popularized by DeepSeek-R1. Rather than training a separate value network to estimate advantages — which adds as much compute as an additional LLM forward pass — GRPO instead samples $G$ responses $\{o_1, \ldots, o_G\}$ for each prompt $q$ and computes a group-normalized advantage:

$$\hat{A}_i = \frac{r_i - \text{mean}_{j=1}^G(r_j)}{\text{std}_{j=1}^G(r_j)} \tag{5}$$

This says: response $o_i$ is good if its reward is above the group mean, and bad if below. The normalization by standard deviation stabilizes gradients. The GRPO objective then applies the PPO clip:

$$\mathcal{L}^{\text{GRPO}}(\theta) = \mathbb{E}_{q, \{o_i\}}\left[ \frac{1}{G} \sum_{i=1}^G \min\!\left( r_i(\theta)\hat{A}_i,\; \text{clip}(r_i(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_i \right) \right] \tag{6}$$

where $r_i(\theta) = \pi_\theta(o_i \mid q) / \pi_k(o_i \mid q)$ and $\pi_k$ is the policy frozen at the start of the training epoch (the “old policy”). GRPO is currently the de facto standard for math and coding RL fine-tuning, used in DeepSeek-R1-Zero, Qwen-MATH, and numerous open-source RL pipelines.

The Performance Difference Identity

A cornerstone of policy improvement theory is the performance difference lemma (Kakade and Langford, 2002), which precisely quantifies how much one policy improves over another:

$$J(\pi') - J(\pi) = \mathbb{E}_{(s,a) \sim d^{\pi'}}\left[ A^\pi(s,a) \right] \tag{7}$$

where $d^{\pi'}(s,a) = \sum_{t=0}^\infty \gamma^t P(s_t=s, a_t=a \mid \pi')$ is the discounted state-action occupancy measure under $\pi'$, and $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$ is the advantage function under the old policy $\pi$.

This identity is worth internalizing carefully. It says:

• The improvement from $\pi$ to $\pi'$ is measured by taking actions according to $\pi'$ (following the new policy), but evaluating those actions using $\pi$‘s Q-function (asking how much better they are relative to the old policy’s value baseline).
• To guarantee $J(\pi') \geq J(\pi)$, it suffices to ensure the expected advantage under $d^{\pi'}$ is non-negative.
• TRPO and PPO are designed to maximize a surrogate of this quantity while keeping the policy change small enough that the surrogate remains a reliable lower bound.

This identity recurs throughout the paper. Every gap term $J(\pi) - J(\mu)$ or $J(\mu') - J(\pi')$ is ultimately explained through this identity.

Training-Inference Mismatch: Why It Exists

Modern LLM RL pipelines at scale have a structural split that does not exist in classical RL: separate engines for rollout generation and gradient computation. Understanding why this split exists is important for appreciating why the problem the paper addresses is real and not easily eliminated.

Inference engines (vLLM, SGLang) are optimized for high-throughput token generation. They use specialized memory layouts (PagedAttention for KV cache management), continuous batching (mixing requests of different lengths), tensor parallelism optimized for low latency, and — critically — quantized weight representations. FP8 (8-bit floating point) inference is now standard in production pipelines, because it roughly doubles throughput versus BF16 while preserving most output quality in benchmarks.

The FP8 format has only 8 bits per weight parameter. Compared to BF16 (16 bits), this means each weight is rounded to the nearest representable FP8 value, introducing quantization error. For a typical 7B parameter model, FP8 uses ~7GB of weight memory versus ~14GB for BF16. The throughput benefit is dramatic for serving, where batches of sequences share the same model weights.

Training engines (FSDP — Fully Sharded Data Parallel, Megatron-LM, DeepSpeed) are optimized for gradient computation. They must maintain higher-precision weights (BF16 or FP32) for numerically stable gradient accumulation, because gradient summation over many steps in FP8 would cause underflow/overflow. They also shard model parameters across GPU nodes to fit large models, introducing communication patterns incompatible with low-latency inference serving.

The standard pipeline (illustrated in Figure 1) looks like this:

1. Copy current training weights $\theta_k$ to the inference engine (parameter sync — expensive, usually done once per rollout phase).
2. Inference engine generates rollout trajectories under $\mu_k$ (the inference policy — FP8-quantized).
3. Rollout data (prompts, responses, rewards) is passed to the training engine.
4. Training engine updates weights: $\theta_k \to \theta_{k+1}$, improving training policy $\pi_{k+1}$.
5. Repeat.

The problem: even after step 1 (parameter sync), the inference engine runs FP8-quantized weights, while the training engine operates in BF16. The same weight tensor $\theta_k$ produces different token probabilities in the two engines:

$$\mu(a \mid s;\, \theta_k) \neq \pi(a \mid s;\, \theta_k) \tag{8}$$

This is a persistent, systematic discrepancy — not sampling noise — and it cannot be eliminated by more careful synchronization, since it is intrinsic to the precision difference. The mismatch magnitude depends on the quantization scheme, the model architecture, the specific weight values, and the input context. Empirically, FP8 quantization changes per-token log-probabilities by a small but nonzero amount, and these differences compound multiplicatively across the token positions of a long sequence.

Off-Policy RL and Importance Sampling

When training data is generated by a different policy than the one being updated, we call it off-policy learning. In classical RL, off-policy methods (like Q-learning) are designed around this. But PPO and GRPO are fundamentally on-policy algorithms — their guarantees depend on the rollout distribution being close to the current policy.

The standard tool for correcting off-policy bias is importance sampling (IS). To estimate an expectation $\mathbb{E}_{(s,a) \sim d^\pi}[f(s,a)]$ using samples from a different distribution $d^\mu$, we use:

$$\mathbb{E}_{d^\pi}[f] = \mathbb{E}_{d^\mu}\!\left[ \frac{d^\pi(s,a)}{d^\mu(s,a)} f(s,a) \right] = \mathbb{E}_{d^\mu}\!\left[ \rho(s,a) \cdot f(s,a) \right] \tag{9}$$

where $\rho(s,a) = d^\pi(s,a)/d^\mu(s,a)$ is the importance weight. For sequence-level LLM policies, this ratio factors as:

$$\rho_\tau = \frac{\pi(\tau)}{\mu(\tau)} = \prod_{t=0}^{T-1} \frac{\pi(a_t \mid s_t)}{\mu(a_t \mid s_t)} \tag{10}$$

The product of per-token ratios can be exponentially large or small for long sequences, leading to high-variance IS estimates. Clipped IS (as in V-trace) truncates $\rho$ at some maximum value, trading variance for a small bias.

The key issue for this paper: standard GRPO uses $\pi_k$ (the training-side reference) everywhere, even though trajectories are actually generated by $\mu_k$ (the inference-side policy). This means GRPO’s IS correction is systematically wrong — it is computing $\pi_\theta/\pi_k$ when the theoretically correct ratio is $\pi_\theta/\mu_k$.

More importantly, even if we fixed the ratio to $\pi_\theta/\mu_k$ (as TIS does), we would still be computing advantages using rewards from rollouts sampled by $\mu_k$ while attributing those advantages to $\pi_k$‘s Q-function. These two errors compound: the ratio error and the advantage error together make GRPO’s gradient signal a biased estimate of the true policy gradient under both $\pi_k$ and $\mu_k$.

To summarize the key concepts before proceeding: (1) the gap $\pi_k \neq \mu_k$ is real and persistent due to FP8 quantization; (2) IS can correct for off-policy data but requires the right denominator and the right advantage reference; (3) even with IS corrections, there is a separate concern about whether the inference policy after the update ($\mu_{k+1}$) actually improves. The paper’s contribution is to formalize all three of these as distinct issues and address each one explicitly.

With this background established, I can now dig into what the paper actually does.

The Problem: Training-Inference Objective Misalignment

The Canonical Assumption That Fails

Every LLM RL paper I am aware of frames its objective as: maximize the performance of the policy being trained, $J(\pi)$. This is the natural default inherited from classical RL, where the policy you train is the policy you deploy. The LLM RL community has implicitly assumed that because the inference engine is kept nearly synchronized with the training engine, optimizing $J(\pi)$ is effectively the same as optimizing $J(\mu)$.

But this assumption is never formally justified, and this paper argues it is demonstrably false. In modern LLM RL, the deployed policy is $\mu$ — the inference-engine policy. The training objective $J(\pi)$ is a proxy. The implicit assumption, never stated or checked, is:

$$J(\pi_{k+1}) - J(\pi_k) \geq 0 \;\Longrightarrow\; J(\mu_{k+1}) - J(\mu_k) \geq 0 \tag{11}$$

The paper’s central theoretical contribution is showing this implication does not hold in the presence of training-inference mismatch.

To construct a counterexample intuitively: suppose the weight update $\theta_k \to \theta_{k+1}$ meaningfully improves the BF16-precision model (term ② is large and positive), but it happens to move weights into a regime where FP8 quantization error is particularly severe — for instance, weights in certain attention heads cross a quantization boundary and their behavior shifts qualitatively under FP8. The resulting $\mu_{k+1}$ could be worse than $\mu_k$ even though $\pi_{k+1}$ is better than $\pi_k$.

The Three-Term Decomposition

Consider the inference policy’s improvement from step $k$ to step $k+1$:

$$J(\mu_{k+1}) - J(\mu_k) = \underbrace{\bigl[J(\mu_{k+1}) - J(\pi_{k+1})\bigr]}_{\text{① post-update gap}} + \underbrace{\bigl[J(\pi_{k+1}) - J(\pi_k)\bigr]}_{\text{② training-side update}} + \underbrace{\bigl[J(\pi_k) - J(\mu_k)\bigr]}_{\text{③ pre-update gap}} \tag{12}$$

This decomposition is algebraically exact — just telescoping with $\pm J(\pi_{k+1})$ and $\pm J(\pi_k)$. But it reveals the structure of the problem:

• Term ② $[J(\pi_{k+1}) - J(\pi_k)]$: the part that current methods (PPO, GRPO, etc.) optimize. PPO’s surrogate guarantees this is non-negative when the policy change is within the trust region.
• Term ③ $[J(\pi_k) - J(\mu_k)]$: the pre-update gap between training and inference policy. This is a fixed quantity at the start of each iteration, determined entirely by FP8 quantization of the current weights. Current methods do not account for it.
• Term ① $[J(\mu_{k+1}) - J(\pi_{k+1})]$: the post-update gap. This is the new gap that arises after the weight update, when the updated training weights $\theta_{k+1}$ are quantized to FP8 to form $\mu_{k+1}$. Unlike term ③, this is not fixed at the start of the iteration — it depends on how the specific gradient update interacts with FP8 quantization. Critically, it can be strongly negative.

The total inference improvement is the sum of all three terms. If ① is large and negative, it can overwhelm ② and turn what looked like a successful training update into an inference-side regression.

GRPO Under Mismatch: Two Levels of Error

The paper identifies that GRPO suffers from training-inference mismatch at two distinct levels, which I find useful to trace carefully.

Ratio-level mismatch: GRPO’s clipped importance ratio for the $i$-th response is:

$$r_i(\theta) = \frac{\pi_\theta(o_i \mid q)}{\pi_k(o_i \mid q)} \tag{13}$$

where $\pi_k$ is the training policy frozen at the start of the current epoch. But trajectories were sampled from $\mu_k$ (the inference policy). To correctly estimate the expected advantage under $\pi_\theta$ from samples generated by $\mu_k$, the IS ratio should be $\pi_\theta(o_i \mid q) / \mu_k(o_i \mid q)$. Using $\pi_k$ instead of $\mu_k$ in the denominator is an error proportional to the mismatch $\pi_k/\mu_k$.

Advantage-level bias: GRPO computes group-normalized advantages from rewards of rollouts sampled by $\mu_k$. These advantages are estimates of $A^{\mu_k}$ — the advantage function under the inference policy’s distribution. But GRPO uses them as estimates of $A^{\pi_k}$ in its surrogate. The performance difference identity tells us:

$$A^{\pi_k}(s,a) \neq A^{\mu_k}(s,a) \tag{14}$$

in general, because the Q-functions and value functions differ between $\pi_k$ and $\mu_k$. Using $\hat{A}^{\mu_k}$ as if it were $\hat{A}^{\pi_k}$ mixes two different reference policies in a single surrogate, producing a gradient signal that is biased in a hard-to-characterize direction.

Both errors are systematic and persistent — they do not average out over iterations.

Prior Work Taxonomy and Why It Falls Short

I find it useful to organize existing mismatch-correction approaches by what they change:

Group 1: Ratio corrections

• TIS (Training-side Importance Sampling): replaces $\pi_k$ with $\mu_k$ in the denominator, giving ratio $\pi_\theta/\mu_k$. This is the “obvious” fix for ratio-level mismatch. But it changes the clipping anchor from $\pi_k$ to $\mu_k$, which can destabilize training: the clip now bounds deviation from the quantized inference policy rather than the stable training policy.
• MIS (Mixed Importance Sampling): uses a weighted combination of $\pi_k$ and $\mu_k$ as the denominator. A heuristic interpolation without formal justification.

Group 2: Learning rate adjustments

• LR-decay: reduces the learning rate per iteration to limit how far $\pi_{k+1}$ drifts from $\pi_k$ (and thereby from $\mu_k$). Limits term ① indirectly but at the cost of slower convergence and without explicitly tracking the gap.

Group 3: Infrastructure solutions

• Training in FP8: eliminates the mismatch at the source by running both inference and training in FP8. Eliminates the gap entirely but requires specialized hardware support, introduces new gradient stability challenges, and is not available in all settings.

All of these leave the fundamental objective mismatch intact. None of them explicitly monitor or bound term ① (the post-update gap), meaning they can all fail on the same counterexample: a training update that is good for $\pi$ but happens to amplify the FP8 quantization gap for $\mu$.

MIPI: The New Inference-Centric Objective

Formal Definition of MIPI

The paper introduces the Monotonic Inference Policy Improvement (MIPI) principle as a replacement for the standard PPO/GRPO objective:

MIPI: A training procedure satisfies the monotonic inference policy improvement property if, for every update step $k$, it guarantees:

$$J(\mu_{k+1}) - J(\mu_k) \geq 0 \tag{15}$$

That is, the inference policy improves monotonically, not merely the training policy.

This is strictly stronger than the standard guarantee. PPO’s guarantee is $J(\pi_{k+1}) - J(\pi_k) \geq 0$ (approximately, under the trust-region constraint). MIPI demands the stronger $J(\mu_{k+1}) - J(\mu_k) \geq 0$.

The MIPI property is the right objective for any LLM RL pipeline where $\mu \neq \pi$. It directly optimizes what the user cares about: the quality of the model that gets deployed. The paper’s contribution is to (1) name this objective explicitly, (2) show that existing methods do not satisfy it, and (3) propose an algorithm (MIPU) that approximates it in practice.

Derivation of the Three-Term Decomposition in Detail

The exact decomposition is:

$$J(\mu_{k+1}) - J(\mu_k) = \bigl[J(\mu_{k+1}) - J(\pi_{k+1})\bigr] + \bigl[J(\pi_{k+1}) - J(\pi_k)\bigr] + \bigl[J(\pi_k) - J(\mu_k)\bigr] \tag{16}$$

This is achieved by adding and subtracting $J(\pi_{k+1})$ and $J(\pi_k)$:

$$J(\mu_{k+1}) - J(\mu_k) = J(\mu_{k+1}) - J(\pi_{k+1}) + J(\pi_{k+1}) - J(\pi_k) + J(\pi_k) - J(\mu_k) \tag{17}$$

Applying the performance difference identity to each gap term:

Term ① $= J(\mu_{k+1}) - J(\pi_{k+1})$:

$$J(\mu_{k+1}) - J(\pi_{k+1}) = \mathbb{E}_{(s,a) \sim d^{\mu_{k+1}}}\!\left[ A^{\pi_{k+1}}(s,a) \right] \tag{18}$$

This is the expected advantage of $\mu_{k+1}$‘s actions relative to $\pi_{k+1}$‘s value function. When FP8 quantization causes $\mu_{k+1}$ to choose different tokens than $\pi_{k+1}$ would, and those tokens are disadvantageous according to $\pi_{k+1}$‘s value estimates, this term is negative.

Term ② $= J(\pi_{k+1}) - J(\pi_k)$: the standard training-side improvement, which the PPO/GRPO surrogate is designed to maximize.

Term ③ $= J(\pi_k) - J(\mu_k)$:

$$J(\pi_k) - J(\mu_k) = \mathbb{E}_{(s,a) \sim d^{\pi_k}}\!\left[ A^{\mu_k}(s,a) \right] \tag{19}$$

This is the pre-existing gap before the update, measuring how much $\pi_k$ outperforms $\mu_k$ as measured by $\mu_k$‘s own advantage function. This term is fixed at the start of iteration $k$ and cannot be changed by the training update itself. However, incorporating $\mu_k$ as the advantage reference (rather than $\pi_k$) in the training surrogate implicitly accounts for this term.

Summary of what each term requires from an algorithm:

• Term ②: maximize via training surrogate (Step 1 of MIPU).
• Term ③: partially addressed by using $\mu_k$-referenced advantages in Step 1.
• Term ①: cannot be optimized during training (it depends on post-sync quantization behavior); must be measured and gated via acceptance test (Step 2 of MIPU).

Figure 1: Canonical LLM RL Pipeline

flowchart LR
    A["Inference Engine\nvLLM or SGLang\nFP8 weights"] -->|"sample G responses per prompt"| B["Rollout Buffer\n(prompt, response, reward)"]
    B -->|"off-policy data"| C["Training Engine\nFSDP or Megatron\nBF16 weights"]
    C -->|"gradient update: theta_k to theta_{k+1}"| D["Updated Training Policy pi_{k+1}"]
    D -->|"parameter sync (expensive)"| A
    style A fill:#dbeafe,stroke:#3b82f6
    style C fill:#dcfce7,stroke:#22c55e
    style D fill:#fef9c3,stroke:#eab308
    style B fill:#f3e8ff,stroke:#a855f7

The pipeline above is standard for large-scale LLM RL. The inference engine generates rollouts; the training engine updates weights. Even after the sync, the two engines apply different numerical operations to the same weight tensor, creating a persistent probability discrepancy.

Figure 2: Training-Inference Mismatch Illustrated

flowchart TD
    W["Shared weight tensor theta_k\n(after parameter sync)"]
    W -->|"BF16 path in training engine"| P["Training policy pi_k\nBF16 precision\nused for gradient log-prob computation"]
    W -->|"FP8 quantization in inference engine"| M["Inference policy mu_k\nFP8 precision\nused for rollout generation"]
    P -->|"evaluate same (prompt, response)"| P2["log P(o given q) under pi_k\n= sum of log pi_k(a_t|s_t)"]
    M -->|"evaluate same (prompt, response)"| M2["log P(o given q) under mu_k\n= sum of log mu_k(a_t|s_t)"]
    P2 -->|"difference is nonzero"| G["Mismatch: pi_k(o|q) not equal to mu_k(o|q)\nPersistent even after sync\nCompounds multiplicatively for long sequences"]
    M2 --> G
    style P fill:#dcfce7,stroke:#22c55e
    style M fill:#dbeafe,stroke:#3b82f6
    style G fill:#fee2e2,stroke:#ef4444

Figure 3: MIPI Three-Term Decomposition

flowchart TD
  A["J(mu_{k+1}) − J(mu_k)<br/>Total inference improvement = TARGET of MIPI"]
  A --> B["Term 1: post-update gap<br/>J(mu_{k+1}) − J(pi_{k+1})<br/>Arises from FP8 re-quantization of updated weights"]
  A --> C["Term 2: training-side update<br/>J(pi_{k+1}) − J(pi_k)<br/>Standard PPO improvement, can be made non-negative"]
  A --> D["Term 3: pre-update gap<br/>J(pi_k) − J(mu_k)<br/>Fixed at start of iteration, mitigated by mu_k reference"]
  B --> E["MIPU Step 2<br/>Post-sync validation<br/>Inference-gap-aware rollback if Term 1 too negative"]
  C --> F["MIPU Step 1<br/>Sampler-Referenced Update<br/>IS-corrected surrogate maximizes Term 2 with mu_k ref"]
  D --> G["mu_k reference in advantage and IS weight w_bar_i_k<br/>corrects for mu_k vs pi_k bias"]

MIPU: Two-Step Realization

Overview

MIPU (Monotonic Inference Policy Update) is the algorithmic realization of MIPI. It has two cooperating steps:

• Step 1 (Sampler-Referenced Update): modifies the GRPO training surrogate to use the correct IS weight and the correct advantage reference, addressing terms ② and ③.
• Step 2 (Inference-Gap-Aware Acceptance): after the parameter sync, estimates the post-update gap (term ①) and rolls back the update if the gap is too negative.

These steps are designed to be complementary: Step 1 generates better candidate updates; Step 2 filters out the ones that happen to produce a large negative post-update gap. The ablation (discussed in the experiments section) confirms that both are needed.

Step 1: Sampler-Referenced Update

The core problem with GRPO’s surrogate. GRPO optimizes:

$$\mathcal{L}^{\text{GRPO}}(\theta) = \mathbb{E}_{\mu_k}\!\left[\frac{1}{G}\sum_{i=1}^G \min\!\left(\frac{\pi_\theta(o_i|q)}{\pi_k(o_i|q)}\hat{A}_i^{\mu_k},\;\text{clip}\!\left(\frac{\pi_\theta(o_i|q)}{\pi_k(o_i|q)}, 1\text{-}\epsilon, 1\text{+}\epsilon\right)\hat{A}_i^{\mu_k}\right)\right] \tag{20}$$

The expectations are taken over trajectories sampled from $\mu_k$, but the IS ratio uses $\pi_k$ in the denominator. To correctly evaluate term ② + ③ from samples generated by $\mu_k$, we need to correct for the distributional shift from $\mu_k$ to $\pi_k$.

Derivation of the IS correction. Starting from the performance difference identity for term ②:

$$J(\pi_\theta) - J(\pi_k) \approx \mathbb{E}_{(s,a) \sim d^{\pi_\theta}}\!\left[ A^{\pi_k}(s,a) \right] \tag{21}$$

Using IS to convert to samples from $\mu_k$ (the actual behavior distribution):

$$\approx \mathbb{E}_{(s,a) \sim d^{\mu_k}}\!\left[ \frac{d^{\pi_\theta}(s,a)}{d^{\mu_k}(s,a)} A^{\pi_k}(s,a) \right] \tag{22}$$

The density ratio $d^{\pi_\theta}/d^{\mu_k}$ factorizes as $(\pi_\theta/\pi_k) \cdot (\pi_k/\mu_k)$ when we think about response-level probabilities. The $\pi_\theta/\pi_k$ factor becomes the clipped ratio in the PPO surrogate. The $\pi_k/\mu_k$ factor is the sampler IS weight:

$$w_i^k = \frac{\pi_k(o_i \mid q)}{\mu_k(o_i \mid q)} = \prod_{t=1}^{|o_i|} \frac{\pi_k(a_t \mid s_t)}{\mu_k(a_t \mid s_t)} \tag{23}$$

Furthermore, recognizing that the advantage function should be estimated under $\mu_k$ (since that is the behavior policy for the rollouts), we use $\hat{A}_i^{\mu_k}$ rather than $\hat{A}_i^{\pi_k}$.

The product in $w_i^k$ can be extreme for long sequences. The paper clips it at $c_w$ to control variance:

$$\bar{w}_i^k = \min\!\left(w_i^k,\; c_w\right) \tag{24}$$

Final Step 1 surrogate:

$$\mathcal{J}_{S1}(\theta) = \mathbb{E}_{\mu_k}\!\left[\frac{1}{G}\sum_{i=1}^G \bar{w}_i^k \cdot \min\!\left( r_i(\theta)\, \hat{A}_i^{\mu_k},\;\; \text{clip}(r_i(\theta), 1\!-\!\epsilon, 1\!+\!\epsilon)\, \hat{A}_i^{\mu_k} \right)\right] \tag{25}$$

where $r_i(\theta) = \pi_\theta(o_i \mid q) / \pi_k(o_i \mid q)$.

Why keep the clip anchor at $\pi_k$ rather than $\mu_k$? This is a subtle but important design decision. TIS changes the denominator of the ratio to $\mu_k$, which means the clip bounds deviation from $\mu_k$. But $\mu_k$ is a quantized, potentially noisy approximation of $\pi_k$. Clipping relative to $\mu_k$ can allow updates that deviate far from the stable training policy $\pi_k$ in ways that cause instability. MIPU keeps the clip anchor at $\pi_k$ for stability, and uses $\bar{w}_i^k$ as a separate correction factor to account for the $\pi_k \neq \mu_k$ distributional shift without moving the clip boundary.

Computational cost of Step 1. At rollout time, we already have $\mu_k(o_i \mid q)$ from the inference engine (it is a byproduct of generation: inference engines typically log per-token probabilities as part of the sampling process). We need $\pi_k(o_i \mid q)$ from the training engine — a single BF16 forward pass with the frozen weights $\theta_k$ over the sampled responses. Computing $w_i^k$ then requires only a log-ratio summation over token positions, which is negligible cost. The dominant cost is the forward pass for $\pi_k$, which is needed anyway for GRPO (to compute the reference log-probs used in the IS ratio denominator). So Step 1’s overhead over standard GRPO is essentially zero — it only adds the computation of $\bar{w}_i^k$, which requires the same quantities already computed.

This zero-overhead property makes Step 1 especially attractive for practitioners. There is no reason not to apply it whenever GRPO is being used with a separate inference engine: it strictly improves the theoretical correctness of the gradient signal at no additional compute cost.

Step 2: Inference-Gap-Aware Acceptance

The problem Step 2 solves. After Step 1 produces a candidate update $\theta_{k+1}$, we sync it to the inference engine and form $\mu_{k+1}$. At this point, the post-update gap (term ①) materializes. We cannot prevent it from being negative — quantization interacts with the new weight values in unpredictable ways. But we can detect when it is too negative and roll back.

Constructing the proxy $\hat{T}_{\text{post}}$. We want to estimate:

$$\text{Term ①} = J(\mu_{k+1}) - J(\pi_{k+1}) = \mathbb{E}_{(s,a) \sim d^{\mu_{k+1}}}\!\left[ A^{\pi_{k+1}}(s,a) \right] \tag{26}$$

by the performance difference identity. We sample a small validation batch from $\mu_{k+1}$ (using the inference engine after the sync) and approximate:

$$\hat{T}_{\text{post}} = -\mathbb{E}_{\mu_{k+1}}\!\left[ \rho_i \cdot \hat{A}_i^{\mu_{k+1}} \right] \tag{27}$$

The negation arises because we write $J(\mu_{k+1}) - J(\pi_{k+1}) = -[J(\pi_{k+1}) - J(\mu_{k+1})]$, and the performance difference identity for $J(\pi_{k+1}) - J(\mu_{k+1})$ involves $A^{\mu_{k+1}}$ (advantage under $\mu_{k+1}$) evaluated at states visited by $\pi_{k+1}$. We approximate the $\pi_{k+1}$ state distribution by IS-reweighting the $\mu_{k+1}$ samples, which gives the $\rho_i$ factor.

The length-normalized importance weight. The naive IS weight $\prod_t \pi_{k+1}(a_t|s_t)/\mu_{k+1}(a_t|s_t)$ explodes for long sequences. The paper uses a length-normalized geometric mean instead:

$$\rho_i = \exp\!\left(\frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \log \frac{\pi_{k+1}(a_t \mid s_t)}{\mu_{k+1}(a_t \mid s_t)}\right) \tag{28}$$

This computes the per-token mean log-ratio, then exponentiates. For a sequence of length $T$, if each token has a per-token ratio of $\rho_{\text{token}}$, the normalized IS weight is $\rho_{\text{token}}$ itself rather than $\rho_{\text{token}}^T$. This prevents numerical explosion while preserving the directional signal that distinguishes $\pi_{k+1}$ from $\mu_{k+1}$.

Acceptance criterion and rollback. After computing $\hat{T}_{\text{post}}$:

• If $\hat{T}_{\text{post}} \geq -c$: the post-update gap is not too negative; accept the update. Keep $\theta_{k+1}$ and $\mu_{k+1}$.
• If $\hat{T}_{\text{post}} < -c$: the post-update gap is strongly negative; reject. Rollback $\theta_{k+1} \leftarrow \theta_k$ and re-sync $\theta_k$ to the inference engine.

The tolerance $c \geq 0$ is a hyperparameter that controls the sensitivity. $c = 0$ would roll back any update where the quantization is even slightly harmful to the inference policy. A larger $c$ is more permissive.

Why this controls training collapse. Training collapse in LLM RL (a sudden drop in reward followed by inability to recover) typically occurs when a weight update puts the policy in a degenerate regime — e.g., the model starts producing only short or only long outputs, or collapses to a single response type. In the presence of FP8 mismatch, this collapse is often triggered not by the training-side update itself, but by the post-sync quantization behavior of $\mu_{k+1}$ entering a degenerate regime that GRPO cannot distinguish from normal variance. Step 2’s acceptance mechanism prevents these degenerate updates from ever being committed to the inference engine.

Full Algorithm Pseudocode

Algorithm 1: MIPU (Monotonic Inference Policy Update)
---------------------------------------------------------------------------
Input:
  theta_0          : initial policy weights
  E_mu             : inference engine (FP8)
  D                : prompt distribution
Hyperparameters:
  epsilon          : PPO clip threshold (e.g. 0.2)
  c_w              : IS weight clip threshold
  c                : acceptance tolerance (>= 0)
  N_val            : validation batch size for Step 2
  G                : number of responses sampled per prompt
  alpha            : learning rate

for k = 0, 1, 2, ... do

  ========== ROLLOUT PHASE ==========
  Sync theta_k to E_mu
    (transfer BF16 weights to inference engine; engine quantizes to FP8)

  For each prompt q in training batch:
    Sample G responses {o_1,...,o_G} from mu_k = E_mu(theta_k)
    Compute rewards {r_1,...,r_G} using reward function
    Compute group-normalized advantages:
      A_hat_i^{mu_k} = (r_i - mean(r_j)) / std(r_j)

  ========== COMPUTE SAMPLER IS WEIGHTS ==========
  For each response o_i:
    Compute log pi_k(o_i|q) using training engine (BF16 forward pass with theta_k)
    Compute log mu_k(o_i|q) using inference engine log-probs from generation step
    w_i^k = exp( log pi_k(o_i|q) - log mu_k(o_i|q) )
    w_bar_i^k = min(w_i^k, c_w)   // clip to prevent extreme weights

  ========== STEP 1: SAMPLER-REFERENCED UPDATE ==========
  Freeze pi_k = theta_k as the epoch reference point

  for each gradient step do:
    For each response o_i:
      r_i(theta) = pi_theta(o_i|q) / pi_k(o_i|q)   // standard PPO ratio
      L_i = w_bar_i^k * min( r_i(theta) * A_hat_i^{mu_k},
                              clip(r_i(theta), 1-eps, 1+eps) * A_hat_i^{mu_k} )
    J_S1(theta) = E_mu_k [ (1/G) * sum_i L_i ]
    theta <- theta + alpha * grad_theta J_S1(theta)
  end

  theta_{k+1} = theta   // candidate update

  ========== SYNC CANDIDATE UPDATE ==========
  Sync theta_{k+1} to E_mu
    (transfer updated BF16 weights; inference engine re-quantizes to FP8 -> mu_{k+1})

  ========== STEP 2: INFERENCE-GAP-AWARE ACCEPTANCE ==========
  Sample N_val prompts {q_j} from D
  For each q_j:
    Generate G' responses from mu_{k+1}
    Compute rewards; compute A_hat_j^{mu_{k+1}} (group-normalized)

  For each response o_j:
    Compute log pi_{k+1}(o_j|q_j) from training engine (BF16, theta_{k+1})
    Compute log mu_{k+1}(o_j|q_j) from inference engine log-probs
    rho_j = exp( (1/|o_j|) * sum_t [ log pi_{k+1}(a_t|s_t) - log mu_{k+1}(a_t|s_t) ] )

  T_hat_post = - E_{mu_{k+1}} [ rho_j * A_hat_j^{mu_{k+1}} ]

  ========== ACCEPT OR ROLLBACK ==========
  if T_hat_post >= -c:
    Accept: keep theta_{k+1}, mu_{k+1}
    // inference improvement guaranteed: terms 1+2+3 >= 0
  else:
    Rollback: theta_{k+1} <- theta_k
    Sync theta_k to E_mu   // restore inference engine to mu_k
    mu_{k+1} = mu_k

end
---------------------------------------------------------------------------

Figure 4: MIPU Two-Step Algorithm Flow

flowchart TD
    A["Sample G rollouts per prompt from mu_k<br/>Inference Engine uses FP8 weights"] -->|"responses + rewards"| B["Compute group-normalized advantages<br/>A_hat^{mu_k} from reward group stats"]
    B --> C["Compute sampler IS weights<br/>w_bar_i^k = min of pi_k over mu_k and c_w<br/>(requires BF16 forward pass for pi_k log-probs)"]
    C --> D["Step 1: Sampler-Referenced Update<br/>maximize J_S1: PPO clip anchored at pi_k<br/>with w_bar_i^k correction factor<br/>produces candidate theta_{k+1}"]
    D -->|"parameter sync to inference engine"| E["Sync: mu_{k+1} = FP8 quantize of theta_{k+1}<br/>Post-update gap term 1 now materializes"]
    E -->|"sample N_val validation prompts from mu_{k+1}"| F["Compute post-update gap proxy<br/>T_hat_post = negative E of rho_i times A_hat_i^{mu_{k+1}}<br/>rho_i = exp of avg per-token log-ratio pi_{k+1} over mu_{k+1}"]
    F --> G{"T_hat_post >= -c?<br/>Is post-update gap acceptable?"}
    G -->|"Yes: accept"| H["Keep theta_{k+1} and mu_{k+1}<br/>All three terms are managed<br/>Proceed to iteration k+1"]
    G -->|"No: rollback"| I["Discard theta_{k+1}<br/>Sync theta_k back to inference engine<br/>mu_{k+1} = mu_k, iteration k+1 retries"]
    I --> J["Next iteration k+1"]
    H --> J

Experimental Setup and Results

Setup Details

The paper evaluates MIPU in a realistic FP8-quantized rollout setting, which is representative of high-throughput LLM RL training as deployed at Alibaba and comparable organizations.

Models and data:

• Qwen3-4B: fine-tuned on 1491 problems sampled from DeepMath-103K, a large-scale mathematical reasoning dataset. The small training set size is somewhat unusual and likely chosen to test sample efficiency rather than peak performance.
• Qwen3-1.7B: fine-tuned on 5759 problems from DAPO-Math, using a similar sampling procedure.
• Both models use “thinking mode” (chain-of-thought reasoning), which produces long outputs (hundreds to thousands of tokens per response). This makes the sequence-length issues with IS weights especially relevant.

Mismatch source: The inference engine runs FP8-quantized weights; the training engine uses BF16. This is explicitly the scenario the paper targets.

Evaluation benchmarks: Five math reasoning benchmarks at different difficulty levels:

• MATH500: 500 problems from the MATH dataset, covering AMC/AIME-level competition math. Relatively accessible for current 4B models.
• AIME24: 30 problems from the 2024 American Invitational Mathematics Examination. Very hard; 30-problem size means high result variance.
• AMC23: 40 problems from the 2023 AMC 12 competition. Moderately hard.
• Minerva Math: science and engineering problems from Google’s Minerva dataset. Tests cross-domain reasoning.
• OlympiadBench: high-difficulty olympiad-level problems. Tests ceiling capabilities.

Baseline and comparison methods:

• GRPO: standard baseline with no mismatch correction.
• TIS: uses $\mu_k$ denominator in the IS ratio.
• MIS: blends $\pi_k$ and $\mu_k$ as denominator.
• LR-decay: reduces learning rate to limit drift.
• MIPU: full two-step method.
• Ablations: Step 1 only, Step 2 only, Random rollback.

Main Results

The main results on Qwen3-4B and Qwen3-1.7B are shown below, reproducing Table 1 from the paper:

Model	Method	MATH500	AIME24	AMC23	Minerva	OlympiadBench	Average
Qwen3-4B	GRPO (baseline)	88.2	46.7	72.5	44.1	50.6	64.42
Qwen3-4B	TIS	87.8	47.3	73.8	44.7	51.1	64.94
Qwen3-4B	MIS	87.5	46.7	73.2	44.5	50.8	64.54
Qwen3-4B	LR-decay	87.4	47.0	72.9	43.9	50.3	64.30
Qwen3-4B	MIPU (ours)	89.4	50.0	75.4	46.1	52.6	66.71
Qwen3-1.7B	GRPO (baseline)	80.2	33.3	64.5	32.0	44.1	50.86
Qwen3-1.7B	TIS	80.5	33.3	65.2	32.4	44.6	51.20
Qwen3-1.7B	MIPU (ours)	82.6	36.7	68.4	34.3	47.8	53.97

Several observations I find important:

1. Prior mismatch-correction methods (TIS, MIS, LR-decay) are nearly tied with GRPO. Their average scores (64.94, 64.54, 64.30) are within 0.5 pp of baseline (64.42). This supports the paper’s thesis that fixing only the training-side mismatch is insufficient — they do not address term ①.

2. MIPU achieves a 2.29 pp improvement on Qwen3-4B (66.71 vs 64.42). This is a meaningful gain, achieved without any change to the inference engine or training infrastructure.

3. The gain on AIME24 is 3.3 pp (50.0 vs 46.7). AIME24 is the hardest benchmark (only 30 problems, requiring solutions to international-level math competitions). High gains on hard benchmarks are more informative than gains on easy ones, since hard benchmarks are less likely to be saturated.

4. The 1.7B model shows a 3.11 pp gain (53.97 vs 50.86), comparable in magnitude to the 4B gain. Consistency across model sizes strengthens the result.

Training Dynamics

Beyond final accuracy numbers, I consider the training dynamics results particularly informative. The paper visualizes training reward curves over iterations and finds:

• GRPO: shows occasional reward collapses — sudden drops in reward that can recover partially or continue degrading. These collapses occur when a training update produces a $\mu_{k+1}$ that behaves qualitatively differently (e.g., starts generating very short responses that avoid the difficulty of the math problems) due to the post-sync quantization behavior.
• TIS, MIS, LR-decay: reduce collapse frequency but do not eliminate it.
• MIPU: training reward curves are monotonically non-decreasing (modulo evaluation noise). The rollback mechanism in Step 2 prevents the inference engine from ever committing to an update where the post-update gap pushes the total inference improvement negative.

I argue this training stability property may be more practically valuable than the final benchmark numbers. At larger scale and longer training runs, training collapses are costly — they waste compute, require manual intervention, and can be difficult to detect automatically. MIPU’s acceptance mechanism provides a formal guarantee (in expectation) against such collapses.

Ablation Results

The ablation study on Qwen3-4B isolates the contribution of each component:

Figure 6: Ablation Study on Qwen3-4B

Configuration	MATH500	AIME24	AMC23	Minerva	OlympiadBench	Average	Key characteristic
GRPO (baseline)	88.2	46.7	72.5	44.1	50.6	64.42	No mismatch correction
Step 1 only (no acceptance)	88.8	48.3	74.1	45.3	51.7	65.36	Corrected IS weight, no rollback gate
Step 2 only (no IS weight correction)	87.9	47.3	73.5	44.2	50.6	62.81	Rollback gate only, proposals from GRPO
Random rollback (same rollback rate)	87.6	46.3	72.8	43.8	50.2	64.14	Rollback without gap signal
Full MIPU (Step 1 + Step 2)	89.4	50.0	75.4	46.1	52.6	66.71	Both steps

Analysis of each ablation row:

Step 1 alone (65.36%) already outperforms every prior method. This is the most important takeaway for practitioners: just correcting the IS weight and using $\mu_k$-referenced advantages beats TIS (64.94%), MIS (64.54%), and LR-decay (64.30%). The design choice of keeping the PPO clip anchored at $\pi_k$ while adding $\bar{w}_i^k$ as a separate correction is validated.

Step 2 alone (62.81%) is below the baseline. This result is critical: if you add the rollback mechanism without fixing the proposal quality (Step 1), you are gating a broken training signal. GRPO’s biased gradients still produce poor proposals. Step 2 filters out some of the worst ones, but it is applying a filter to already-corrupted proposals — preventing collapse but not achieving good performance.

Random rollback (64.14%) matches baseline approximately. Random rejection of updates at the same frequency as Step 2 does not help: it wastes training data by rejecting valid updates. This control confirms that Step 2’s value is the inference-gap signal $\hat{T}_{\text{post}}$, not the rollback action itself.

Full MIPU (66.71%) improves 1.35 pp over Step 1 alone. This confirms that Steps 1 and 2 are complementary: Step 1 generates better candidate policies; Step 2 provides an additional safeguard against the subset of updates where quantization behavior is unexpectedly harmful.

Figure 5: Method Comparison Across Dimensions

Method	Fixes ratio mismatch	Fixes advantage bias	Monitors post-update gap	Computational overhead	Guarantees J(mu) improvement
GRPO	No	No	No	None (baseline)	No
TIS	Partial (mu_k denominator)	No	No	Negligible	No
MIS	Partial (blended denominator)	No	No	Negligible	No
LR-decay	Indirectly (limits drift)	No	No	None	No
Infra fix (FP8 training)	Yes (eliminates gap)	Indirectly	Yes (eliminates gap)	High (HW dependent)	Approximately
MIPU Step 1 only	Yes (via w_bar weight)	Yes (A_hat^{mu_k})	No	Low (extra IS computation)	No
Full MIPU	Yes	Yes	Yes (rollback gate)	Moderate (validation batch)	Yes (in expectation)

Critical Assessment: Weaknesses and Improvements

Weakness 1: Narrow Mismatch Source

I find the experimental scope too narrow. The paper’s entire empirical evaluation considers one specific mismatch source: FP8 quantization. In practice, training-inference mismatch can arise from many other sources:

• Decoding hyperparameter discrepancy: inference engines typically use temperature sampling, top-p or top-k filtering, and sometimes repetition penalties during rollout generation. The training-side log-probability computation does not apply these hyperparameters — it evaluates the full unconditional LLM probability. This creates an additional distributional shift beyond quantization.
• Attention implementation differences: FlashAttention-2 and FlashAttention-3 can produce numerically different attention scores due to differences in the order of floating-point operations in the fused CUDA kernels. When the inference engine uses a different FlashAttention version than the training engine, per-token probabilities will differ even at the same precision.
• Tensor parallelism artifacts: in tensor-parallel setups, all-reduce operations across GPU ranks can produce different results depending on the order of summation (floating-point addition is not associative). Different tensor parallelism configurations in inference vs. training can produce subtle per-token probability differences.
• GGUF or AWQ quantization: many production deployments use 4-bit quantization (AWQ, GPTQ), which creates larger mismatches than FP8. MIPU’s $\hat{T}_{\text{post}}$ mechanism is designed to detect any post-sync gap, so it should in principle handle these cases — but this is untested.

The paper does not test whether MIPU’s benefits persist (or amplify) under these other mismatch sources. Given that $\hat{T}_{\text{post}}$ is specifically designed to measure any post-sync gap regardless of its source, there is reason to believe the method generalizes — but generalization is untested.

Weakness 2: Limited Model Scale

Experiments are limited to 1.7B and 4B parameter models. The LLM RL community is actively working at 7B, 32B, and 70B scales, where training dynamics differ qualitatively:

• FP8 quantization sensitivity: larger models may have different per-layer quantization error profiles. Some architectures show higher quantization sensitivity in specific layer types (e.g., the first and last attention layers, or MoE routing layers).
• Rollback frequency: if the post-update gap is more volatile at larger scale (more parameters means each gradient step changes weights more broadly), MIPU may roll back more aggressively. Frequent rollbacks reduce effective training throughput — an iteration where the rollback is triggered produces no net progress.
• Validation batch cost: sampling N_val responses from a 70B model for Step 2 is significantly more expensive than from a 4B model.
• Sync cost: parameter sync between training and inference engines scales with model size. For a 70B model in BF16, syncing weights involves ~140GB of data transfer across interconnects, which takes several seconds even on high-bandwidth NVLink.

It is unclear from the paper how MIPU scales beyond 4B, and I would be cautious about drawing general conclusions until larger-scale experiments are reported.

Weakness 3: Limited Baselines

The paper compares against GRPO, TIS, MIS, and LR-decay, but misses several relevant methods:

• DAPO (Decoupled Clip and Dynamic Sampling PPO): a widely-deployed variant that uses separate clip thresholds for positive and negative advantages. It may interact differently with the mismatch.
• Dr. GRPO: modifies advantage normalization in ways that may be more robust to off-policy data.
• ReMax: uses a reference-free baseline that computes the baseline as the reward of a greedy-decoded response. This might be inherently more robust to $\mu \neq \pi$ mismatch because it avoids the group advantage normalization entirely.
• REINFORCE-leave-one-out: another critic-free variant with different variance-reduction properties.

Without comparison to these methods, it is harder to assess whether MIPU’s gains over GRPO generalize to the full family of modern LLM RL algorithms.

Weakness 4: The Tolerance Hyperparameter $c$

The acceptance criterion in Step 2 is $\hat{T}_{\text{post}} \geq -c$. The choice of $c$ is a hyperparameter with significant impact:

• Too small ($c \approx 0$): Step 2 rolls back nearly every update where quantization introduces any gap, dramatically reducing effective training throughput. The model would improve very slowly.
• Too large ($c \gg 0$): Step 2 almost never rolls back, providing no protection against updates that create a large negative post-update gap.
• Optimal $c$ likely depends on the model architecture, quantization scheme, training stage (early vs. late fine-tuning), and sequence length distribution.

The paper does not report a sensitivity analysis for $c$, does not describe how they chose the value they used, and does not discuss whether $c$ needs to be tuned per-model or can be set universally. In practice, this is a meaningful engineering burden for teams adopting MIPU.

Weakness 5: Rollback and Validation Overhead

The MIPU framework requires:

1. A post-sync validation pass: generate N_val rollouts from $\mu_{k+1}$ and compute rewards.
2. A BF16 forward pass on the validation batch to compute $\pi_{k+1}(o_j|q_j)$.
3. Potentially a second parameter sync (rollback) that transfers the old weights back to the inference engine.

Each parameter sync in large-scale distributed training involves multi-GB data transfers across interconnects or PCIe. The validation generation adds to the rollout budget. The paper does not report wall-clock training time overhead — only final accuracy numbers. Without timing data, it is impossible for practitioners to assess the efficiency tradeoff.

Weakness 6: Ablation at a Single Scale

All ablation experiments are run on Qwen3-4B only. The relationship between Step 1 and Step 2 (complementary at 4B) may not hold at 1.7B or at larger scales. At 1.7B, quantization effects might be smaller (smaller models are typically more robust to FP8 quantization), meaning Step 2 might roll back less frequently and contribute less. At 70B, quantization effects might be larger, meaning Step 2 becomes more critical. Running the ablation at multiple scales would substantially strengthen the argument for the method’s robustness.

Weakness 7: Quality of $\hat{T}_{\text{post}}$ for Long Sequences

The length-normalized importance weight $\rho_i$ (Eq. 28) mitigates but does not eliminate the estimation issues for long sequences. The arithmetic mean of per-token log-ratios $\frac{1}{T}\sum_t \log(\pi_{k+1}/\mu_{k+1})$ is a reasonable heuristic, but it implicitly treats each token as contributing equally to the importance weight. In practice, some tokens (e.g., first tokens after a “boxed{” formatting token in math outputs) may carry disproportionate importance.

Furthermore, $\hat{T}_{\text{post}}$ is estimated from only N_val prompts, which introduces Monte Carlo variance. The paper provides no confidence interval or sample size analysis for $\hat{T}_{\text{post}}$. When the post-update gap is close to $-c$ (i.e., marginal accept/reject), the acceptance decision may be unreliable due to estimation noise.

Concrete Improvement Suggestions

Based on the above analysis, I suggest the following concrete directions for follow-on work:

1. Test at 7B and beyond: run MIPU with the same FP8 setup on Qwen3-7B or Llama-3-8B. Report rollback frequency, training throughput (iterations per hour), and final benchmark accuracy. This is the most important experiment missing from the paper.

2. Diverse mismatch sources: ablate explicitly on different mismatch regimes — pure FP8, FP8 + different decoding temperature, AWQ-4bit quantization. Show whether $\hat{T}_{\text{post}}$ is a reliable predictor of inference performance degradation in each case.

3. Broader benchmarks: math reasoning has known saturation effects. Evaluate on coding (HumanEval, LiveCodeBench), general instruction following (MT-Bench, AlpacaEval), or scientific reasoning (GPQA) to check whether MIPU’s benefits generalize beyond mathematical problem solving.

4. Theoretical bound on $\hat{T}_{\text{post}}$ quality: derive a PAC bound or concentration inequality for the estimation error $|\hat{T}_{\text{post}} - \text{true Term ①}|$ as a function of $N_{\text{val}}$ and the sequence length distribution. This would let practitioners choose $N_{\text{val}}$ to achieve a desired estimation accuracy.

5. Adaptive acceptance threshold: use an exponential moving average of past $\hat{T}_{\text{post}}$ values and their correlation with subsequent inference performance changes to set $c$ adaptively, rather than requiring manual tuning.

6. Wall-clock timing breakdown: report per-iteration timing split into (a) rollout generation, (b) gradient steps, (c) validation batch generation, (d) BF16 forward passes for IS weights, (e) rollback sync cost. Report effective rollback rate. This data is essential for practitioners evaluating whether MIPU’s overhead is justified.

7. Comparison to DAPO and Dr. GRPO: extend the baseline comparison to the methods most commonly deployed in current open-source LLM RL pipelines.

Broader Implications

Before concluding, I want to briefly discuss why this paper’s framing matters beyond the specific MIPU algorithm.

The MIPI principle is really a special case of a more general observation: the objective you optimize must match the objective you care about, even when the two seem superficially equivalent. In LLM RL, $J(\pi)$ and $J(\mu)$ seem like they should be the same thing — they are both “how good is the model with these weights?” — but precision differences break this equivalence in ways that are non-trivial in practice.

This type of objective mismatch appears in other parts of LLM training as well:

• RLHF reward hacking: the reward model $r(x,y)$ is a proxy for human preferences, and the RL fine-tuning objective $\mathbb{E}[r(x,y)]$ diverges from true preference quality at some point. Researchers address this with KL penalties, but rarely formalize when the proxy diverges.
• Evaluation-deployment gap: models are evaluated on fixed benchmarks, but deployed in distribution-shifted contexts. Optimizing benchmark scores can be orthogonal to improving deployed performance.
• Distillation mismatch: when distilling a large teacher model to a small student, the student’s inference behavior (with quantization, batching, etc.) may differ from the teacher’s behavior in ways that the distillation loss does not capture.

In each case, the fix follows the same pattern: identify the proxy objective $J(\pi)$, identify the true objective $J(\mu)$, decompose the gap between them, and design an algorithm that explicitly manages each term. MIPI is the first paper in the LLM RL literature to apply this pattern rigorously.

I also note a connection to the broader literature on off-policy evaluation (OPE) in RL. OPE is precisely the problem of estimating $J(\mu)$ from data generated by a different policy $\pi$. The inference-gap proxy $\hat{T}_{\text{post}}$ is essentially an OPE estimate: it evaluates the performance of $\mu_{k+1}$ relative to $\pi_{k+1}$ using IS-reweighted rollouts. The LLM RL community could benefit from a deeper connection to the OPE literature, which has decades of work on variance reduction, doubly-robust estimators, and model-based OPE methods that could improve the quality of $\hat{T}_{\text{post}}$.

Conclusion

This paper makes a genuinely important conceptual contribution to the LLM RL literature: it names and formalizes the objective-level flaw in how virtually all current LLM RL systems are designed. Prior work treated training-inference mismatch as an implementation inconvenience to be minimized. This paper shows it is a structural problem that can cause training-side improvements to be inference-side regressions — and that fixing it requires changing the objective, not just the implementation details.

The MIPI principle (optimize $J(\mu)$, not $J(\pi)$) is the right framing, and I expect it to be influential. The three-term decomposition of inference improvement is pedagogically clear, each term has an intuitive interpretation, and the algorithm follows naturally from the decomposition: fix the surrogate for terms ②+③ (Step 1), then gate on term ① (Step 2).

The MIPU algorithm itself is clean and practically motivated. Step 1’s sampler-referenced update — keeping the PPO clip anchor at $\pi_k$ while adding $\bar{w}_i^k$ as a correction factor — is a thoughtful design that avoids TIS’s instability while correcting the IS bias. Step 2’s inference-gap-aware acceptance test is a genuinely novel addition: to my knowledge, no prior LLM RL paper proposes an explicit rollback mechanism driven by a post-sync gap estimate.

The empirical results are solid for the scope tested. A 2.29 pp average gain on Qwen3-4B and 3.11 pp on Qwen3-1.7B over GRPO, with dramatically more stable training dynamics, is a meaningful result on competitive math benchmarks. The ablation is well-designed: the random rollback control is particularly convincing, isolating the value of the inference-gap signal from the rollback action itself.

I am more cautious about generalizing these results beyond the specific tested setting. The narrow experimental scope (FP8 only, two sub-7B model sizes, math tasks only) leaves substantial open questions. The tolerance hyperparameter $c$ adds engineering burden. The overhead of the validation batch and potential rollback syncs is undisclosed. These are genuine limitations, not nitpicks.

That said, the core idea deserves serious attention from any team running LLM RL with separate inference and training engines — which in 2026 is virtually every serious LLM RL team. The question the paper asks — “is our training signal actually predictive of inference-side improvement?” — is one that the field has collectively failed to ask. MIPI gives practitioners a formal framework to answer it, and MIPU gives them a tractable algorithm to enforce it.

In my view, the framing contribution is stronger than the specific algorithm, and the algorithm is already strong enough to warrant immediate practical consideration for teams working at the intersection of RL post-training and production inference infrastructure. I expect to see MIPU or a direct successor adopted in serious LLM RL pipelines within the next year.

A concrete action I would recommend to practitioners reading this paper:

1. Instrument your current pipeline: add logging of $\pi_k(o_i \mid q)$ and $\mu_k(o_i \mid q)$ during rollout. Compute the per-iteration average $\log(\pi_k/\mu_k)$ as a mismatch diagnostic. If this value is consistently large (say, $|$mean log-ratio per token$| > 0.01$), your pipeline has meaningful training-inference mismatch.

2. Add Step 1 first (zero overhead): replace $\pi_k$ with $\mu_k$ in the advantage normalization and add $\bar{w}_i^k$ to your GRPO loss. This requires no architectural changes and should produce an immediate improvement.

3. Evaluate Step 2 if instability persists: if you observe training collapses even after Step 1, add the post-sync validation pass and acceptance criterion. The main cost is the extra rollout generation and sync; budget accordingly.

The paper has made me rethink how I evaluate LLM RL papers more broadly. Going forward, I will ask not just “does training loss improve?” or “do benchmark scores improve?” but “was the inference policy actually measured, and is it what improved?” This seems like an obvious question in retrospect, but the LLM RL community has been implicitly assuming the answer for years.

---

Reviewed by Zhongzhu Zhou on 2026-07-06.