RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents

Review date: 2026-07-07 Review author: Zhongzhu Zhou Paper reviewed: RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents Paper authors: Qiang Liu, Taian Guo, Ruizhi Qiao, Xing Sun arXiv: https://arxiv.org/abs/2607.04713 Status/Venue: Preprint, July 2026, Tencent YouTu Lab

Short Answer

RSPO is a training framework for multi-turn LLM agents that solves a specific and painful problem: how do you give a language model useful feedback when the only ground-truth signal you have is a single binary outcome at the very end of a long interaction sequence? The answer is not to rely purely on that sparse signal, and not to replace it with a learned dense signal (which leads to reward hacking), but to use the dense signal as a temporary scaffold — an exploration engine — and then immediately discard it when computing the final update. The model that explores under dense rewards generates diverse trajectories; the model that learns from those trajectories is updated using only ground-truth outcomes. Reward-swap is the name for this handoff.

The name is slightly misleading at first read. “Reward-Swap” does not mean swapping good rewards for bad rewards, or swapping rewards between agents. It means that the reward type used for exploration (dense process rewards) is swapped for a different reward type for the actual policy update (outcome rewards). The two roles of reward — as an exploration signal and as a learning target — are separated and assigned different reward types. The swap is the assignment of reward type to role.

In practice, RSPO wraps any proximal policy gradient method (PPO or GRPO) with a cyclic two-phase loop. In phase one, the current policy becomes an explorer by temporarily training on dense per-step rewards; in phase two, the agent trains on the explorer’s collected trajectories re-labeled with outcome rewards, blended with its own on-policy samples. Two technical contributions make this work cleanly: a generalized clipping mechanism that handles the distributional gap between the on-policy agent and the off-policy explorer, and a reward-based sampling strategy for the replay buffer. On ALFWorld and WebShop benchmarks using Qwen2.5-1.5B and 7B models, RSPO consistently improves over GRPO, GiGPO, and PPO baselines by margins ranging from 1.6 to 14.9 percentage points in success rate.

The paper is from Tencent YouTu Lab, released as a preprint in July 2026. It is technically clean, well-ablated in the areas it covers, and honest about its scope. If you are working on multi-turn agent training with LLMs in the 1–10B range, this is a paper worth engaging with carefully — not just reading the abstract, but understanding why each component of RSPO is necessary and what would break if you removed it.

Prerequisites

Before diving into RSPO, you need a working mental model of several concepts that the paper treats as background knowledge. I will go through each one carefully, because the interaction between them is exactly where RSPO’s design choices become legible.

The Markov Decision Process (MDP)

An MDP is a mathematical framework for sequential decision making. At each discrete time step $t$, the world is in some state $s_t$. An agent observes this state and chooses an action $a_t$ from some action space. The world transitions to a new state $s_{t+1}$ according to a transition function $p(s_{t+1} | s_t, a_t)$, and the agent receives a scalar reward $r_t = r(s_t, a_t)$. The agent’s goal is to find a policy $\pi$ — a mapping from states to distributions over actions — that maximizes expected cumulative discounted reward:

$$\mathbb{E}_{\tau \sim \pi}\left[\sum_{t=0}^{\infty} \gamma^{t} r_t\right] \tag{MDP}$$

where $\tau = (s_0, a_0, s_1, a_1, \ldots)$ is a trajectory and $\gamma \in [0,1)$ is a discount factor that down-weights future rewards. The discount factor has two roles: mathematical (ensures the sum converges) and practical (rewards the agent for solving problems quickly rather than eventually).

For LLM agents, the state $s_t$ typically contains the original task instruction plus the full history of actions and environment responses up to step $t$. The action $a_t$ is the text output the model produces at step $t$ — a search query, a navigation command, a purchase click, or a reasoning statement. The transition function is the environment: a web simulator, a text game engine, a code interpreter.

Value Functions and the Credit Assignment Problem

To improve a policy, we need to know not just what total reward we got, but which actions were responsible for it. This is the credit assignment problem, and it is harder than it sounds.

Define the state-value function $V^\pi(s)$ as the expected total discounted reward starting from state $s$ and following policy $\pi$:

$$V^\pi(s) = \mathbb{E}_\pi\left[\sum_{t=0}^{\infty} \gamma^t r_t \;\middle|\; s_0 = s\right] \tag{V}$$

Define the action-value function $Q^\pi(s,a)$ as the expected total reward starting from $s$, taking action $a$, then following $\pi$:

$$Q^\pi(s,a) = \mathbb{E}_\pi\left[\sum_{t=0}^{\infty} \gamma^t r_t \;\middle|\; s_0 = s,\, a_0 = a\right] \tag{Q}$$

The advantage function $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$ measures how much better or worse action $a$ is compared to the average action at state $s$. An advantage greater than zero means “this action was better than average”; less than zero means “this action hurt.”

These quantities are central because good policy gradient methods do not tell the agent “maximize reward” in a brute-force way; they tell it “take actions where the advantage is positive.” That distinction matters enormously for training stability.

REINFORCE: The Simplest Policy Gradient

The most direct way to train a policy with gradient descent is the REINFORCE estimator. The key insight is that the gradient of expected reward with respect to policy parameters $\theta$ can be written as:

$$\nabla_\theta \mathbb{E}_\tau[R(\tau)] = \mathbb{E}_\tau\left[R(\tau) \nabla_\theta \log \pi_\theta(\tau)\right] \tag{REINFORCE}$$

In words: run the current policy, collect a trajectory, compute its total reward $R(\tau)$, then push the log-probability of every action in that trajectory up or down proportionally to whether the trajectory succeeded or failed. This is mathematically correct but has very high variance — one lucky trajectory can dominate the gradient estimate and send the policy in a completely wrong direction.

Variance reduction is achieved by subtracting a baseline $b$ (usually $V^\pi(s_t)$) from the reward:

$$\nabla_\theta J(\theta) \approx \mathbb{E}_\tau\left[\sum_t (R(\tau) - b) \nabla_\theta \log \pi_\theta(a_t | s_t)\right] \tag{REINFORCE-baseline}$$

The baseline does not change the expected gradient (it has zero mean under the expectation) but dramatically reduces variance. In the limit, using the optimal baseline $b = V^\pi(s_t)$ replaces $R(\tau)$ with the advantage $A^\pi(s_t, a_t)$.

Temporal Difference Learning and the Bellman Equation

Before reaching PPO, it helps to understand how $V^\pi$ is actually estimated in practice. The naive approach is Monte Carlo: run the policy to the end of each episode, then set $V^\pi(s_t) \approx$ the average total return observed across all episodes that started from $s_t$. This is unbiased but high variance, and it requires waiting until the episode terminates before updating anything — impractical for long trajectories.

Temporal difference (TD) learning offers an alternative. It exploits the recursive structure of value functions, known as the Bellman equation:

$$V^\pi(s_t) = \mathbb{E}_\pi\left[r_t + \gamma V^\pi(s_{t+1})\right] \tag{Bellman}$$

The Bellman equation says the value of a state equals the expected immediate reward plus the discounted value of the next state. Rather than waiting until the end of the episode, TD learning performs a one-step update every timestep:

$$V(s_t) \leftarrow V(s_t) + \alpha_{\text{lr}}\left(r_t + \gamma V(s_{t+1}) - V(s_t)\right) \tag{TD-update}$$

The quantity $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$ is called the TD error. It measures how surprised the agent is by the actual transition: if the transition was better than expected (positive $\delta_t$), the value estimate for $s_t$ is nudged upward. This one-step bootstrapping trades a small amount of bias for a large reduction in variance and allows online learning during the episode.

The advantage estimate in actor-critic methods uses a one-step TD error as an approximation:

$$\hat{A}(s_t, a_t) \approx r_t + \gamma V(s_{t+1}) - V(s_t) \tag{TD-advantage}$$

This is called the TD advantage estimate. More sophisticated variants use $n$-step returns or Generalized Advantage Estimation (GAE) to interpolate between the high-variance Monte Carlo and the biased one-step TD estimate:

$$\hat{A}^{\text{GAE}}_t = \sum_{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l} \tag{GAE}$$

where $\lambda \in [0,1]$ is a decay parameter. When $\lambda=0$, GAE reduces to one-step TD; when $\lambda=1$, it reduces to Monte Carlo returns. PPO implementations typically use GAE with $\lambda \approx 0.95$.

Actor-Critic Architecture

Actor-Critic methods separate the policy (actor) from the value function estimator (critic). The actor is the policy $\pi_\theta$ that takes actions; the critic is a learned value function $V_\phi$ that evaluates states. During training, the critic bootstraps value estimates for advantage computation, and the actor uses those advantages to update its parameters.

In the context of LLM agents, the actor is the LLM itself (parameterized by the billions of weights in the transformer). The critic is typically an additional small network (or a separate head on the LLM’s representation) that predicts scalar values. PPO requires a critic; GRPO eliminates the need for one by estimating advantages through within-group reward normalization. This is a key reason GRPO is popular for LLM training: it avoids the engineering overhead of maintaining and training a separate value function network at LLM scale.

The tradeoff is that GRPO’s advantage estimates are less precise than a well-trained critic’s estimates, because they depend on the variance within a finite sample group rather than on a learned approximation to $V^\pi$. For tasks with very sparse rewards where whole groups often succeed or fail together, this imprecision becomes a significant problem — exactly the setting RSPO is designed to address.

KL Regularization and Reference Models

One more prerequisite: understanding why LLM training uses KL divergence as a regularizer. When fine-tuning a large pre-trained model with RL, the policy update can collapse the model’s diversity. A few updates with high-reward trajectories can cause the model to assign near-zero probability to everything except the exact phrasing that got high reward — a phenomenon sometimes called mode collapse. The resulting model is brittle and fails on inputs that differ even slightly from the training distribution.

The KL term in GRPO’s objective, $\beta \mathbb{D}_{\mathrm{KL}}(\pi_\theta \| \pi_{\text{ref}})$, prevents this by penalizing divergence from the frozen reference model $\pi_{\text{ref}}$ (the SFT checkpoint). The hyperparameter $\beta$ controls the strength of this penalty. Larger $\beta$ keeps the policy close to the reference and reduces reward-hacking risk, at the cost of limiting how much the policy can improve. Smaller $\beta$ allows larger improvements but risks instability.

In practice, choosing $\beta$ well is non-trivial and depends on the quality of the SFT checkpoint. RSPO inherits this hyperparameter from its base algorithm (GRPO or PPO) without modification.

Proximal Policy Optimization (PPO)

REINFORCE in its basic form has another problem: the policy update can be catastrophically large. If one update sends the policy into a bad region of parameter space, future samples will be drawn from that bad region, and recovery is slow or impossible. PPO solves this with the concept of a trust region.

PPO uses importance sampling to allow multiple gradient steps on a single batch of collected data (improving sample efficiency) while constraining how far the new policy can drift from the policy that collected the data. Define the importance sampling ratio:

$$r_t(\theta) = \frac{\pi_\theta(a_t | s_t)}{\pi_{\theta_{\text{old}}}(a_t | s_t)} \tag{IS-ratio}$$

where $\pi_{\theta_{\text{old}}}$ is the policy that collected the current batch. The unclipped objective is simply $r_t(\theta) A_t$, but PPO clips this ratio to prevent large updates:

$$\mathcal{J}_{\text{PPO}} = \mathbb{E}_t\left[\min\!\left(r_t(\theta) A_t,\; \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) A_t\right)\right] \tag{PPO}$$

The clip function keeps $r_t(\theta)$ in the interval $[1-\epsilon, 1+\epsilon]$. When the advantage is positive (this action was good), the objective is maximized, but only up to the point where $r_t(\theta) = 1+\epsilon$ — beyond that, gradients are zeroed. When the advantage is negative, the objective is minimized, but only down to $r_t(\theta) = 1-\epsilon$. This two-sided constraint keeps the updated policy close to the data-collection policy.

The key intuition behind the $\min$ in the PPO objective: without the $\min$, the clipped version alone would still allow updates in the wrong direction (making a bad action more likely). The $\min$ ensures the objective is always a lower bound on the unclipped objective, making it a pessimistic estimate that is safe to optimize.

GRPO: Group Relative Policy Optimization

GRPO is a variant of PPO designed specifically for scenarios where we want to evaluate multiple candidate outputs for the same input — exactly the setting of LLM fine-tuning. For a given task $x$, GRPO samples a group of $N$ trajectories $\{\tau_1, \ldots, \tau_N\}$ from the old policy and uses within-group reward normalization to compute advantages:

$$A(\tau_i) = \frac{R(\tau_i) - \text{mean}(\{R(\tau_j)\}_{j=1}^N)}{\text{std}(\{R(\tau_j)\}_{j=1}^N)} \tag{GRPO-A}$$

This normalization is important: it makes the advantage measure relative within the group, so a trajectory that got reward 1 when everyone else got reward 0.5 gets a positive advantage, while the same trajectory in a group where everyone succeeded gets a near-zero advantage. In practice this stabilizes training because the scale of the advantage signal adapts to the current difficulty of the task.

The full GRPO objective is:

$$\mathcal{J}_{\text{GRPO}} = \mathbb{E}_{x \sim p(X),\,\{\tau_i\} \sim \pi_{\theta_{\text{old}}}}\!\left[\frac{1}{NT}\sum_{i=1}^{N}\sum_{t=1}^{T} \min\!\left(r_{i,t}(\theta)\,A_{i,t},\; \text{clip}(r_{i,t}(\theta), 1\!\pm\!\epsilon)\,A_{i,t}\right)\right] - \beta\,\mathbb{D}_{\mathrm{KL}}(\pi_\theta \| \pi_{\text{ref}}) \tag{GRPO}$$

where $r_{i,t}(\theta) = \frac{\pi_\theta(a_t^i | x, s_t^i)}{\pi_{\theta_{\text{old}}}(a_t^i | x, s_t^i)}$ is the importance ratio for action $a_t^i$ in trajectory $i$ at step $t$. The KL divergence term $\beta \mathbb{D}_{\mathrm{KL}}(\pi_\theta \| \pi_{\text{ref}})$ prevents the policy from drifting too far from a frozen reference model $\pi_{\text{ref}}$ (typically the SFT checkpoint), guarding against catastrophic forgetting and reward hacking.

The per-step indexing $\frac{1}{NT}\sum_{i,t}$ is significant: GRPO assigns the same trajectory-level advantage $A_{i,t} = A(\tau_i)$ to every action in that trajectory. This is the right choice when you only have an outcome reward (you cannot attribute per-step credit), but it is also GRPO’s fundamental weakness: if two trajectories reach different intermediate states and only one ultimately succeeds, the per-step advantage cannot distinguish which early actions were actually responsible.

Why Sparse Outcome Rewards Are Hard for GRPO

Consider a 20-step ALFWorld navigation task. The agent must pick up an object, examine it under a lamp, and place it in a container — in the right order. With outcome-only rewards, every action in a successful trajectory gets an advantage of roughly $+1$ and every action in a failed trajectory gets roughly $-1$. The gradient update reinforces the entire successful trajectory uniformly. But some of those 20 actions were exploratory and irrelevant; reinforcing them wastes model capacity and adds noise.

More concretely, there are two failure modes:

First, when all $N$ trajectories in a group either all succeed or all fail, the within-group advantages are all zero (the standard deviation is zero or the reward is constant). No gradient flows at all. This is increasingly likely at the extremes of training — when the model has learned to almost always succeed or when the task is genuinely hard and the model almost always fails. RSPO’s authors call this the “unchosen successful trajectories” problem: if the model happens to sample $N$ failing trajectories but has learned behaviors that could succeed, none of those successful trajectories appear in the batch, and no learning happens.

Second, GRPO with outcome rewards generates a limited diversity of trajectories. Because the policy is trained entirely on its own rollouts, it can converge to a small basin of attraction — a narrow set of strategies that sometimes work. It misses the large space of strategies it has never tried. This is the exploration problem, and it is particularly acute in text-based environments where the action space is combinatorially large.

To quantify the exploration problem: an LLM generating a 20-token action at each of 20 steps in ALFWorld has a nominal action space of $|V|^{20 \times 20}$ where $|V| \approx 32{,}000$ is the vocabulary size. Of course, the effective action space is much smaller because most sequences are grammatically or semantically invalid — but even the effective action space of meaningful navigation commands, object descriptions, and interaction verbs is large enough that any policy trained on $N=16$ trajectories per batch is sampling an infinitesimally small fraction of the relevant space at each training step. The policy’s probability distribution over this space is highly non-uniform, concentrated around phrasings that appeared in the SFT training data, and gradient updates that reinforce successful trajectories narrow this distribution further over time. RSPO’s exploration phase temporarily widens the distribution by temporarily optimizing for a different (dense) signal, then transfers the successful trajectories back to the narrowing policy.

Reward Hacking and Dense Process Rewards

The natural solution to sparse rewards is to train a model that predicts rewards at every step — a process reward model (PRM). Instead of waiting until the end to know if the trajectory succeeded, the agent gets feedback at every action: “this search query moved you closer to the target product.”

Dense rewards fix the credit assignment problem but introduce a new one: reward hacking. A neural process reward model is an approximation of the true per-step value. If you train the agent exclusively on that approximation, it will eventually find policy trajectories that maximize the approximation while completely diverging from the true objective. The model learns to exploit the reward model’s errors rather than to actually solve the task. This is not a theoretical concern — the RSPO authors show it empirically, and I will discuss that evidence in detail in the ablation section.

The tension between outcome rewards (correct but sparse) and process rewards (dense but approximation-prone) is the central problem that RSPO addresses.

What the Paper Does

RSPO proposes a cyclic two-agent framework that uses dense process rewards as an exploration scaffold and outcome rewards as the ground truth signal for final learning. The core insight is that you do not have to choose between dense and sparse rewards: you can use dense rewards to expand the distribution of trajectories available for training, then strip those rewards off and re-label the trajectories with outcome rewards before training the final policy. The reward swap — using one reward type for exploration and a different type for learning — is the key idea from which the method takes its name.

The paper makes three technical contributions: (1) the cyclic reward-swap loop that alternates between an explorer trained on dense rewards and a learner trained on outcome rewards, (2) a generalized clipping mechanism for the PPO/GRPO objective that correctly handles off-policy data from the explorer, and (3) a reward-based trajectory sampling strategy for the replay buffer that maximizes the quality of off-policy data without introducing training instability. Together, these components address trajectory diversity, off-policy correction, and sample selection — the three practical obstacles to making reward swapping work.

One aspect of the paper’s framing that I find important is what RSPO does not claim. It does not claim to train a better process reward model; it does not claim to eliminate the need for a process reward model; it does not claim to solve the sparse reward problem in general. It claims a specific, bounded improvement: by using dense rewards as a temporary behavioral modifier for one agent while keeping another agent’s learning signal ground-truth, you can get both the exploration benefit of dense rewards and the correctness of outcome rewards. That modesty of scope is, I think, a sign of intellectual honesty — the authors have identified a real problem and built a targeted solution rather than overclaiming.

The Multi-Turn RL Problem in Depth

Formalizing the Agent Task as an MDP

In the RSPO framework, multi-turn LLM agent tasks are formalized as MDPs where $s_t$ is the concatenation of the task instruction $x$ with all preceding actions and observations: $s_t = [x; a_1; o_1; a_2; o_2; \ldots; a_{t-1}; o_{t-1}]$. The action $a_t$ is a text string output by the LLM. The observation $o_t$ is the environment’s response — a text game’s feedback, a web page’s content, a code interpreter’s output.

The policy $\pi_\theta$ is an LLM parameterized by $\theta$. The probability of a trajectory $\tau$ under $\pi_\theta$ is:

$$\pi_\theta(\tau | x) = \prod_{t=1}^{T} \pi_\theta(a_t | s_t) \tag{Traj-Prob}$$

The objective is to maximize expected discounted cumulative reward:

$$\mathcal{J}(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta(\cdot|x)}\left[\sum_{t=1}^{T} \gamma^{t-1} r_t\right] \tag{Obj}$$

In the environments RSPO studies — ALFWorld and WebShop — the natural reward structure has exactly one non-zero reward: $r_T = R(\tau)$ at the final step (success or failure). All intermediate rewards are zero. This is the sparse outcome reward setting.

Why GRPO Struggles in This Setting

GRPO’s within-group normalization creates a specific failure pattern for multi-turn tasks. Consider training on ALFWorld with group size $N=8$. If the current policy succeeds on 6 out of 8 trajectories, the 2 failing trajectories get negative advantages and the 6 successful ones get mildly positive advantages. This seems fine, but look more carefully at what happens to the 6 successful trajectories: they all receive essentially the same advantage signal regardless of how they succeeded, which early actions were suboptimal, or which steps were wasted. Every step in a successful trajectory is treated identically. The agent cannot learn to prefer trajectory $A$ (which succeeded efficiently in 8 steps) over trajectory $B$ (which succeeded inefficiently in 18 steps) if both have the same outcome reward.

Now consider the harder case: the policy is in an early phase of training where only 0 or 1 out of 8 trajectories succeed. With 1 success out of 8, the advantage for the successful trajectory is approximately $+2.6$ (since the normalized advantage of one success versus seven failures with reward 0 has mean $\approx 0.125$ and std $\approx 0.33$). The single successful trajectory carries the entire gradient update. If that trajectory happened to contain some unusual actions that worked by coincidence, those actions get reinforced strongly. The model can overfit to that one lucky trajectory and fail to generalize.

With 0 successes, no gradient flows at all — every advantage is exactly zero because every reward is the same. Training stalls completely.

The Diversity Problem

There is a deeper structural problem beyond the gradient signal quality. When the policy trains only on its own rollouts, it explores the state-action space defined by its current parameters. Early in training, when the policy is weak, the on-policy distribution is heavily concentrated in low-quality regions of trajectory space. The policy is essentially sampling from a narrow distribution of strategies and receiving mostly negative feedback — but the strategies that would succeed are elsewhere in the space, and the policy will not find them through local gradient steps.

This is the exploration-exploitation dilemma, well-studied in tabular and continuous RL but particularly hard in the discrete, high-dimensional space of text sequences. Every new word in the LLM’s action changes the trajectory in ways that are hard to anticipate, and the policy’s probability mass is typically concentrated on familiar phrasings that may not transfer to new task instances.

RSPO’s solution to this problem is indirect: rather than building an explicit exploration mechanism into the policy gradient update, it creates a separate agent (Agent B) whose sole purpose is to explore more broadly, then mines Agent B’s trajectories for diverse demonstrations of success that Agent A would never have found on its own.

The Connection to Single-Turn RLHF

It is worth pausing here to contrast multi-turn agent RL with the single-turn RLHF setting that most LLM practitioners are more familiar with. In single-turn RLHF (e.g., training a chatbot to give better answers), the MDP has a single step: the prompt is the state, the full generated response is the action, and the reward is a human preference score or reward model output. There is no multi-step trajectory; the discount factor is irrelevant; and credit assignment is trivial because there is only one action to credit.

GRPO was originally developed for this single-turn setting, and it works extremely well there. The within-group normalization across $N$ responses to the same prompt gives stable and informative advantage estimates, because the response quality varies enough within the group to generate meaningful gradients. The exploration problem is also mild in single-turn settings: the LLM’s natural diversity (through temperature sampling) is sufficient to explore the space of possible responses to a prompt.

Multi-turn agent tasks break both of these favorable properties. The trajectory is long (10–40+ steps), so reward is sparse and credit assignment is hard. The per-step action space is the entire text generation distribution, which is combinatorially larger than the space of discrete choices in classic RL environments. And the exploration problem is severe: the policy must generate not just a diverse response, but a diverse sequence of responses that coherently guides the agent through a long-horizon task. These distinctions are why methods that work well for single-turn RLHF — like basic GRPO with outcome rewards — underperform on multi-turn agent tasks, and why RSPO’s architecture is specifically tailored to the multi-turn setting.

Outcome-Only vs. Dense-Only: A Spectrum of Failure Modes

It helps to think of training reward design as a spectrum with two failure modes at the extremes:

• Outcome-only (left extreme): signal is exact but sparse. Training is slow because most gradient updates are zero or noisy. The model eventually converges if the task success rate is not too low, but the path is sample-inefficient and sensitive to initialization.

• Dense-only (right extreme): signal is dense but approximate. Training is fast initially and the model learns quickly. But the model eventually learns to exploit the approximation, causing the true success rate to plateau or collapse while the dense reward score continues to rise. This is reward hacking, and it is especially dangerous because the training signal gives no warning that it is happening.

RSPO occupies the middle: use dense rewards for exploration (so training is not stuck) but use outcome rewards for updates (so the model cannot hack the dense reward model). The cyclic structure ensures neither failure mode can dominate.

RSPO: The Reward-Swap Framework

System Overview

Figure 1: RSPO system overview — the cyclic Agent A → Agent B → replay buffer → Agent A loop.

flowchart LR
    A["Agent A\n(π_θ, outcome reward)"] -->|"k dense-reward\ntraining steps"| B["Agent B\n(π_dense, explorer)"]
    B -->|"collect trajectories\nτ₁,...,τₘ"| Env["Environment\n(ALFWorld / WebShop)"]
    Env -->|"outcomes R(τ)"| Label["Re-label with\noutcome rewards"]
    Label -->|"store in buffer"| D["Replay Buffer D\n(reward-based sampling)"]
    D -->|"off-policy batch\nD_off (α)"| Mix["Mixed Batch\nD_mix = D_on ∪ D_off"]
    A -->|"on-policy rollouts\nD_on (1-α)"| Mix
    Mix -->|"generalized clipping\nPPO/GRPO update"| A

The loop has a natural rhythm: Agent A temporarily becomes Agent B by training under dense rewards for $k$ steps. Agent B is not a separate model — it is the same parameters after a short detour of dense-reward training. After exploring and collecting trajectories, the framework discards the dense-reward updates and uses those trajectories (re-labeled with outcome rewards) to improve Agent A.

A common confusion when first reading the paper: “doesn’t training Agent B on dense rewards for $k$ steps and then discarding those weights waste compute?” The answer is no, for two reasons. First, $k=3$ is a small number of gradient steps — the compute cost is modest. Second, the value comes not from the weights Agent B develops but from the trajectories Agent B generates. Even imperfect dense-reward training moves Agent B’s sampling distribution toward higher-quality trajectory regions, and those diverse trajectories are the actual output of Phase 1. Think of Phase 1 as a temporary behavioral modification for the purpose of data collection, not as a model checkpoint you care about.

The Dense Process Reward Model

Before describing the algorithm, we need to understand the dense reward model that powers exploration. RSPO uses a lightweight MLP attached to the final hidden layer of a pre-trained LLM ($\pi_{\text{pre}}$, specifically Llama-3.2-3B-Instruct in all experiments). For a state-action pair $(s_t, a_t)$, the hidden representation $h_t$ is extracted from the LLM’s last layer, passed through the MLP, and compressed to a scalar via $\tanh$ to ensure the output lies in $(-1, 1)$:

$$\hat{r}_t = \tanh\!\left(\text{MLP}(h_t)\right), \quad h_t = f_{\pi_{\text{pre}}}(s_t, a_t) \tag{5}$$

The total predicted reward for a trajectory of length $T$ is simply the sum of per-step rewards:

$$\hat{R} = \sum_{t=1}^{T} \hat{r}_t \tag{6}$$

The reward model is trained to minimize mean squared error against ground-truth outcome rewards $R_i \in [0,1]$:

$$\mathcal{L}_P = \frac{1}{N} \sum_{i=1}^{N} \left(\hat{R}_i - R_i\right)^2 \tag{7}$$

Figure 2: Dense process reward model architecture.

flowchart LR
    st["State s_t\n+ Action a_t"] --> LLM["Pre-trained LLM\n(Llama-3.2-3B-Instruct)\nfrozen or shared"]
    LLM --> ht["Hidden State h_t\n(last layer rep.)"]
    ht --> MLP["MLP Layer\n(trainable)"]
    MLP --> tanh["tanh activation\noutput ∈ (−1, 1)"]
    tanh --> rt["Step Reward r̂_t"]
    rt --> Sum["Sum over t=1..T"]
    Sum --> Rhat["Trajectory Reward R̂\n= Σ r̂_t"]
    Rhat --> Loss["MSE Loss vs R_i ∈ [0,1]\nL_P = (1/N) Σ(R̂_i − R_i)²"]

Three design choices are worth noting here. First, the $\tanh$ squashing is deliberate — without it, step rewards could accumulate to arbitrary magnitudes and dominate the training signal. Second, using a separate Llama-3B model as the backbone (rather than the agent model itself) decouples the reward model’s quality from the agent’s improvement, preventing a feedback loop where a degraded agent produces bad hidden representations that corrupt the reward model. Third, training on outcome rewards with an MSE objective means the reward model learns the average per-step contribution to success — a reasonable proxy for step quality, even if imperfect.

The Full Training Loop

Figure 3: Training pipeline — Phase 1 (dense reward exploration) and Phase 2 (outcome reward learning).

flowchart TD
    Init["Initialize:\nπ_θ = π_base, D = ∅"]
    Init --> Loop["RSPO Loop"]

    Loop --> P1["Phase 1: Dense Reward Training → Agent B"]
    P1 --> P1a["Set π_dense = π_θ\n(copy current weights)"]
    P1a --> P1b["for step = 1,...,k:\n  Sample x ~ p(X)\n  Generate (1-α)B trajectories via π_dense\n  Compute dense rewards r̂_t via PRM\n  Update π_dense via P (PPO/GRPO)"]
    P1b --> P1c["Collect trajectories τ via π_dense\nRe-label with outcome rewards R(τ)\nAdd to replay buffer D"]
    P1c --> P2["Phase 2: Outcome Reward Training → Agent A"]

    P2 --> P2a["for step = 1,...,k:\n  Generate D_on (on-policy, size (1-α)B)\n  Sample D_off from D (size αBN)\n  D_mix = D_on ∪ D_off"]
    P2a --> P2b["Update π_θ via P on D_mix\nusing outcome rewards\nGeneralized clipping for D_off"]
    P2b --> P2c["Reset D ← ∅"]
    P2c --> Loop

The algorithm in pseudocode, expanding Algorithm 1 from the paper:

Input: π_base, proximal PG method P, dense reward model R,
       task distribution p(X), group size N, batch size B, steps k, ratio α
Initialize: π_θ = π_base, D = ∅

for each RSPO loop do

  ── Phase 1: Dense reward training ──────────────────────────────────────
  Set π_dense = π_θ                    # branch off a copy
  for step = 1,...,k do
    Sample x ~ p(X)                    # draw a task
    Generate (1-α)B trajectories via π_dense   # explore with dense policy
    Compute dense rewards using PRM    # r̂_t for each step
    Update π_dense using P             # any proximal PG method works
  Collect trajectories τ via π_dense   # final diversity harvest
  Re-label τ with outcome rewards R(τ) # REWARD SWAP: discard dense labels
  Store in replay buffer D             # save for phase 2

  ── Phase 2: Outcome reward training ────────────────────────────────────
  for step = 1,...,k do
    Sample x ~ p(X)
    Generate D_on of size (1-α)B via π_θ   # on-policy rollouts
    Sample D_off of size αBN from D         # off-policy from buffer
    D_mix ← D_on ∪ D_off                   # mixed batch
    Update π_θ using P on D_mix             # Eq. 1 objective
      (on-policy term: standard GRPO clip)
      (off-policy term: generalized clip)
  Reset D ← ∅                              # clear buffer for next loop

return π_θ

Line-by-line explanation of the key decisions:

Set π_dense = π_θ: Phase 1 starts from the current Agent A weights. This ensures the explorer is always at least as good as the current agent, not a random or stale policy. It also means exploration improves as training progresses.

for step = 1,...,k: The loop count $k$ is a hyperparameter balancing exploration quality and reward hacking risk. Too small $k$: Agent B barely improves under dense rewards and generates trajectories similar to on-policy data — no diversity gain. Too large $k$: Agent B overfits the dense reward model and begins reward hacking — the trajectories look good under $\hat{R}$ but are not actually successful by ground truth.

Compute dense rewards using PRM: The process reward model assigns a scalar $\hat{r}_t$ to each step. GRPO/PPO then treats these as environment rewards for the phase-1 update.

Re-label τ with outcome rewards R(τ): This is the reward swap. The trajectories were generated by a policy trained on dense rewards, but their labels are erased and replaced with the ground-truth outcome reward. If the trajectory succeeded, every step gets an advantage signal derived from $R(\tau) = 1$; if it failed, from $R(\tau) = 0$. The exploration strategy (dense) is decoupled from the learning signal (outcome).

D_off sample of size αBN from D: The replay buffer holds trajectories from Agent B. The sampling ratio $\alpha$ controls how much off-policy data blends into the training batch. The authors find $\alpha = 1/8$ works well, with $\alpha = 1/16$ marginally better on WebShop.

Reset D ← ∅: The buffer is cleared after every loop. This prevents stale trajectories from dominating future updates as the policy continues to improve. It is a deliberate choice against a persistent replay buffer. As Agent A improves through Phase 2 training, the trajectories in the buffer — which were generated by Agent B at the start of the current loop — become increasingly off-policy relative to the updated Agent A. Using them in the next loop without re-weighting would introduce importance sampling errors that the generalized clipping cannot fully correct. Resetting forces fresh exploration in every loop, keeping the off-policy gap bounded.

The Total Objective

The training objective for Agent A in Phase 2 mixes on-policy and off-policy terms:

$$\mathcal{J}_{\text{Total}}(\pi_\theta) = (1 - \alpha)\,\mathcal{J}_{\text{On}}(\pi_\theta) + \alpha\,\mathcal{J}_{\text{Off}}(\pi_\theta) \tag{1}$$

The on-policy term is standard GRPO/PPO with outcome rewards — exactly the same as training without RSPO:

$$\mathcal{J}_{\text{On}} = \mathbb{E}_{x \sim p(X),\,\{\tau_i\} \sim \pi_{\theta_{\text{old}}}}\!\left[\frac{1}{NT}\sum_{i=1}^{N}\sum_{t=1}^{T} \min\!\left(r^{\text{on}}_{i,t}(\theta)\,A^{\text{on}}_{i,t},\;\text{clip}\!\left(r^{\text{on}}_{i,t}(\theta), 1\!\pm\!\epsilon\right)A^{\text{on}}_{i,t}\right)\right] - \beta\,\mathbb{D}_{\mathrm{KL}}(\pi_\theta \| \pi_{\text{ref}}) \tag{3}$$

where $r^{\text{on}}_{i,t}(\theta) = \frac{\pi_\theta(a_t^i | x, s_t^i)}{\pi_{\theta_{\text{old}}}(a_t^i | x, s_t^i)}$.

Generalized Clipping for Off-Policy Data

The off-policy term requires more care. The trajectories in the replay buffer were generated by Agent B (policy $\pi_B$), not by $\pi_{\theta_{\text{old}}}$. Standard PPO clipping assumes the data was collected by $\pi_{\theta_{\text{old}}}$, so the clipping interval $[1-\epsilon, 1+\epsilon]$ is centered at 1. When the data comes from $\pi_B$, the correct center point is $\frac{\pi_{\theta_{\text{old}}}(a|s)}{\pi_B(a|s)}$, not 1.

Figure 4: Standard vs. generalized clipping — why the clip center shifts for off-policy data.

flowchart LR
    subgraph Standard["Standard Clipping (on-policy)"]
        direction TB
        sc1["Data from: π_θ_old"]
        sc2["Ratio: r = π_θ / π_θ_old"]
        sc3["At π_θ = π_θ_old: r = 1"]
        sc4["Clip range: [1−ε, 1+ε]\ncentered at 1"]
        sc1 --> sc2 --> sc3 --> sc4
    end
    subgraph Generalized["Generalized Clipping (off-policy)"]
        direction TB
        gc1["Data from: π_B (Agent B)"]
        gc2["Ratio: r = π_θ / π_B"]
        gc3["At π_θ = π_θ_old: r = π_θ_old / π_B ≠ 1"]
        gc4["Clip range: [π_θ_old/π_B − ε, π_θ_old/π_B + ε]\ncentered at π_θ_old/π_B"]
        gc1 --> gc2 --> gc3 --> gc4
    end
    Standard -->|"off-policy gap"| Problem["Standard clip wrongly\nzeroes valid gradients\nwhen π_θ_old/π_B ≠ 1"]
    Generalized --> Solution["Generalized clip\npreserves gradient signal\nfor all valid updates"]

The generalized off-policy objective is:

$$\mathcal{J}_{\text{Off}} = \mathbb{E}_{x,\tau \sim \mathcal{D}}\!\left[\frac{1}{NT}\sum_{i=1}^{N}\sum_{t=1}^{T} \min\!\left(r^{\text{off}}_{i,t}(\theta)\,A^{\text{off}}_{i,t},\;\text{clip}\!\!\left(r^{\text{off}}_{i,t}(\theta),\;\frac{\pi_{\theta_{\text{old}}}(a|s)}{\pi_B(a|s)}\!\pm\!\epsilon\right)A^{\text{off}}_{i,t}\right)\right] - \beta\,\mathbb{D}_{\mathrm{KL}}(\pi_\theta \| \pi_{\text{ref}}) \tag{4}$$

where $r^{\text{off}}_{i,t}(\theta) = \frac{\pi_\theta(a_t^i | x, s_t^i)}{\pi_B(a_t^i | x, s_t^i)}$.

To understand why this matters, it helps to revisit what importance sampling is doing in the first place. When we collect data from distribution $q$ (here, $\pi_B$) but want to estimate expectations under distribution $p$ (here, $\pi_\theta$), importance sampling reweights each sample by $p/q$:

$$\mathbb{E}_{x \sim p}[f(x)] = \mathbb{E}_{x \sim q}\!\left[\frac{p(x)}{q(x)} f(x)\right] \tag{IS}$$

This is unbiased, but the variance of the estimator can be extremely large if $p$ and $q$ differ significantly (large importance weights dominate). Clipping the importance ratio bounds the variance at the cost of introducing bias. Standard PPO clips $\pi_\theta / \pi_{\theta_{\text{old}}}$ around 1 because when the data is on-policy ($\pi_{\theta_{\text{old}}} = q$), the ratio is 1 at the start of the gradient step and deviates from 1 only due to the policy update. Generalized clipping shifts the center to $\pi_{\theta_{\text{old}}} / \pi_B$ to account for the pre-existing distributional gap between $\pi_{\theta_{\text{old}}}$ and $\pi_B$ — the gap that exists before any gradient step is even taken.

To understand why this matters, consider a concrete example. Suppose $\pi_{\theta_{\text{old}}} / \pi_B = 0.5$ for some action — meaning Agent A assigns half the probability to this action that Agent B did. With standard clipping centered at 1, the ratio $r^{\text{off}} = \pi_\theta / \pi_B$ would be clipped to $[1-\epsilon, 1+\epsilon]$ even for the on-policy value $\pi_\theta = \pi_{\theta_{\text{old}}}$. Since $\pi_{\theta_{\text{old}}} / \pi_B = 0.5$ is outside $[1-\epsilon, 1+\epsilon]$ for typical $\epsilon = 0.2$, the gradient for this action would be zeroed even when updating from $\pi_{\theta_{\text{old}}}$ — a mathematically incorrect suppression of valid information. Generalized clipping shifts the interval to $[0.5 - \epsilon, 0.5 + \epsilon]$, correctly preserving gradient signal around the actual on-policy ratio.

In implementation terms, applying generalized clipping requires storing the per-token log-probabilities from both $\pi_{\theta_{\text{old}}}$ and $\pi_B$ for each off-policy sample. The $\pi_B$ log-probabilities must be stored when the trajectory is collected (before Agent B’s weights are overwritten by Phase 2 updates), making replay buffer management slightly more complex than for standard GRPO. This is a low-level but important detail: any implementation that forgets to store $\pi_B$‘s log-probabilities at collection time cannot correctly compute the generalized clip center during training.

Replay Buffer Sampling Strategy

Not all trajectories from Agent B are equally useful. The authors evaluate three sampling strategies for selecting off-policy data from the replay buffer:

• Random: trajectories selected uniformly at random
• Variance-based: trajectories selected based on high reward variance within their groups (similar to active learning)
• Reward-based: trajectories selected in order of descending outcome reward (highest first)

Reward-based sampling dominates strongly (70.6% SR vs. 63.0% SR for variance-based vs. 12.8% SR for random on WebShop — Table 2 in the paper). The intuition is clear in hindsight: the most valuable off-policy data is successful trajectories that the current policy would not generate on its own. Random or variance-based selection includes many failures, which add noise without providing additional exploration signal beyond what on-policy sampling already covers.

Connection to Experience Replay in Classic RL

The RSPO replay buffer shares conceptual ancestry with experience replay in DQN-style deep RL, but with an important difference. In DQN, the replay buffer stores transitions from previous versions of the policy, and the Q-function is updated on samples from this buffer to decorrelate updates and improve sample efficiency. The critical design choice there is also sampling strategy: uniform random replay, prioritized replay (sampling high-error transitions more frequently), or ranked replay.

RSPO’s replay buffer is more specialized: it stores complete trajectories (not individual transitions), its source is a different policy ($\pi_B$) not just an older version of the same policy, and its sampling criterion is outcome reward rather than TD error. But the underlying intuition — maintaining a diverse set of past experiences to prevent the optimizer from myopically over-fitting to recent data — is directly analogous. The RSPO paper does not draw this connection explicitly, which I think is a missed opportunity for situating the work within the broader RL literature.

Three Training Paradigms Compared

Figure 5: Outcome-only, dense-only, and RSPO compared side by side.

graph TD
    O1["Outcome-Only GRPO"] --> O2["Sample N trajectories"]
    O2 --> O3["Scalar reward R at end"]
    O3 --> O4["Sparse gradient, slow learning"]
    O4 --> O5["No hacking, poor exploration"]
    D1["Dense-Only Training"] --> D2["Sample N trajectories"]
    D2 --> D3["Per-step PRM rewards"]
    D3 --> D4["Rich gradient, fast learning"]
    D4 --> D5["Reward hacking, SR collapse"]
    R1["RSPO Reward-Swap"] --> R2["Agent B explores with dense rewards"]
    R2 --> R3["Re-label with outcome R"]
    R3 --> R4["Agent A trains on outcome R"]
    R4 --> R5["Rich exploration, no hacking"]

Experiments and Results

Environments

ALFWorld is a text-based embodied agent benchmark built on the TextWorld engine, which maps a 3D household simulator (ALFRED) to a purely text interface. The agent receives natural language task descriptions (“examine the pencil under the desklamp”) and must issue text commands to manipulate objects and navigate rooms. There are six task types: Pick (find and pick up object), Look (examine an object under a light source), Clean (wash an object in a sink), Heat (heat an object in a microwave), Cool (cool an object in a refrigerator), and Pick2 (pick up two objects). Success is binary: either the agent completes the task correctly or it does not. The benchmark tests compositional reasoning, multi-step planning, and the ability to recover from environment feedback across 20–40 step trajectories.

WebShop is a simulated e-commerce environment where the agent must search for and purchase products matching a natural language specification (e.g., “I need a medium-sized red backpack with laptop pocket under $50"). The agent interacts with a simplified web interface, executing search queries and clicking through product pages. The reward has two components: a continuous score in$[0,1]$ measuring how well the purchased product matches the specification, and a binary success rate (SR) measuring exact matches above a threshold. WebShop tests information retrieval, multi-criterion matching, and sequential decision making over 5–15 step interactions.

Baseline Methods

The paper compares RSPO against four baselines:

• GRPO: the standard group relative policy optimization with outcome rewards, the most direct comparison
• GiGPO: Group-in-Group Policy Optimization, a recent method that introduces hierarchical grouping within GRPO to improve gradient estimation. Tested both with standard normalization (w/std) and without (w/o std)
• PPO: the classic proximal policy optimization baseline with value function estimation
• SPEAR: a concurrent method that uses a process-level reward approach but with larger group sizes (N=32) and more training epochs (350 vs. 150)

RSPO is applied as a wrapper: RSPO+GRPO, RSPO+GiGPO, RSPO+PPO. This is an important design decision — RSPO is not a new base algorithm but an outer loop that improves any proximal PG method.

Key Results

Figure 6: Summary of key experimental results — RSPO gains over baselines.

graph LR
    A1["ALFWorld GRPO: 69.0"] --> A2["RSPO+GRPO: 74.7 +5.7pp"]
    A3["ALFWorld GiGPO: 88.8"] --> A4["RSPO+GiGPO: 90.4 +1.6pp"]
    A5["ALFWorld PPO: 76.6"] --> A6["RSPO+PPO: 85.2 +8.6pp"]
    W1["WebShop GRPO: 46.1"] --> W2["RSPO+GRPO: 50.0 +3.9pp"]
    W3["WebShop PPO: 54.4"] --> W4["RSPO+PPO: 69.3 +14.9pp"]

The 1.5B model results show consistent gains across all baselines and both environments. The most striking number is RSPO+PPO on WebShop: a 14.9 percentage point improvement from 54.4% to 69.3% success rate. This is not a marginal improvement — it is the difference between a system that works roughly half the time and one that works roughly two-thirds of the time.

For the 7B model, the gains are smaller in absolute terms (2.37% average improvement on ALFWorld, 3.96% on WebShop) but still consistent. This is a common pattern in RL fine-tuning: larger models have more representational capacity and can learn more efficiently from sparse signals alone, leaving less room for exploration-based gains. The 1.5B model benefits more because its initial policy is weaker and has more to gain from diverse trajectory data.

For the 7B model, the gains from RSPO are smaller but still consistent, averaging 2.37 percentage points on ALFWorld and 3.96 on WebShop. This diminishing-returns pattern with model scale has a plausible explanation: larger models produce higher-quality on-policy trajectories by default, because their representations better capture the task structure and their language generation is more precise. The on-policy diversity is already higher for 7B than for 1.5B, leaving less room for Agent B’s exploration to add novel trajectories that the base policy would never find. Conversely, at 1.5B, the on-policy distribution is more constrained and more sensitive to initialization, creating a larger gap between what the policy currently does and what it could do with more exploratory guidance.

The SPEAR comparison is nuanced. SPEAR reports higher absolute numbers but uses N=32 vs. RSPO’s N=16 and 350 training epochs vs. 150. When the authors control for these factors by running RSPO and SPEAR under identical conditions, RSPO outperforms SPEAR on WebShop SR (66.4% vs. 63.1%). This is an important methodological point: comparing methods that use different amounts of compute is not a fair comparison, and the paper deserves credit for noting it explicitly.

Per-Task Analysis on ALFWorld

ALFWorld’s six task types are not equally difficult, and the breakdown matters for understanding where RSPO’s gains come from. The Look (examine-in-light) and Pick2 (two-object pickup) tasks are generally harder because they require a longer sequence of coordinated sub-tasks: the agent must find the light source, bring the object to it, and then perform the examination, all while tracking both objects in a potentially large room. These are exactly the tasks where sparse outcome rewards give the weakest per-step credit assignment signal — the agent might succeed on all sub-tasks except the final placement and receive reward 0 for the entire trajectory.

I would expect RSPO’s gains to concentrate in the harder task types (Look, Pick2) rather than the simpler ones (Clean, Heat, Cool), where even GRPO converges quickly because task completion requires fewer interdependent steps and the probability of a successful trajectory is higher. Unfortunately, the paper does not break down results by task type in Table 1, which is an omission I consider a genuine weakness (more on this in the Critical Assessment).

GiGPO Comparison

GiGPO already achieves 88.8% SR on ALFWorld (1.5B) — a very high number — and RSPO+GiGPO adds only 1.6 percentage points (to 90.4%). This small margin reflects the law of diminishing returns: when a baseline is already near the upper end of performance, there is less room for improvement. More interestingly, GiGPO itself is a method designed to improve trajectory diversity through hierarchical grouping, so it partly addresses the same problem RSPO is solving. The fact that RSPO still improves over GiGPO suggests the reward-swap exploration mechanism adds something beyond what grouping tricks alone can provide.

WebShop Score vs. SR Discrepancy

WebShop reports both a continuous score (measuring how well the purchased product matches the specification on a scale of 0 to 1) and a binary success rate (SR, measured as score above a threshold, typically 0.5). These two metrics can tell different stories. RSPO+PPO achieves 69.3% SR versus PPO’s 54.4% SR — a 14.9 percentage point improvement. But if the continuous score improvement were smaller, it would suggest RSPO is primarily helping the model cross the binary threshold for borderline cases rather than genuinely improving the quality of product selection.

Looking at the score numbers in Table 1 (which the paper reports but the discussion underemphasizes), the pattern is consistent: score improvements track SR improvements proportionally. This means RSPO is not just a threshold-crossing trick — it is genuinely improving the quality of the agent’s product selection. The exploration diversity appears to help the model discover strategies that more accurately match multi-criterion specifications, not just strategies that happen to score above 0.5.

Ablation Studies: What Makes RSPO Work

Off-Policy Mixing Ratio $\alpha$ (Table 3)

The $\alpha$ parameter controls what fraction of the training batch comes from the replay buffer. RSPO tests four values on WebShop (RSPO+GRPO, 1.5B):

$\alpha$	Score	SR (%)
1/16	85.1	67.2
1/8	85.0	66.4
1/4	77.4	54.7
1/2	79.1	60.9

The pattern is a sharp cliff around $\alpha = 1/4$. Below that point (1/16, 1/8), performance is excellent and nearly identical. Above it (1/4, 1/2), performance drops substantially.

I find this result interesting for what it implies about the off-policy data quality. At $\alpha = 1/8$, about 12.5% of the training batch comes from Agent B’s trajectories. This is a small contamination of the on-policy distribution, just enough to introduce diversity without disrupting the policy gradient’s locality assumptions. At $\alpha = 1/4$, the off-policy data begins to dominate, and the distribution gap between Agent B and the current Agent A is large enough to cause instability despite the generalized clipping correction.

The fact that $\alpha = 1/16$ slightly outperforms $\alpha = 1/8$ (67.2% vs. 66.4%) while $\alpha = 1/4$ drops to 54.7% suggests a highly non-linear sensitivity. Small $\alpha$ is safe and beneficial; large $\alpha$ is risky. The default of $\alpha = 1/8$ is a reasonable choice that is robust to this non-linearity.

Loop Step Count $k$ (Table 4)

The $k$ parameter controls how many gradient steps Agent B takes under dense rewards before collecting exploration trajectories. Results on WebShop (1.5B):

$k$	Score	SR (%)
1	81.0	65.6
3	85.0	66.4
5	78.5	61.7
10	82.3	61.7

The inverted-U pattern is the clearest signal in all the ablations. With $k=1$, Agent B barely moves away from Agent A — it generates trajectories that look essentially like on-policy data with minimal diversity gain. With $k=3$, Agent B has explored enough to find non-obvious strategies while still being grounded in Agent A’s distribution. With $k=5$ and $k=10$, Agent B has begun to overfit the dense reward model: its trajectories look good under $\hat{R}$ but correspond to reward-hacked behaviors that do not transfer well when re-labeled with ground-truth outcome rewards.

This $k$ sensitivity is one of RSPO’s practical limitations. The optimal $k$ likely depends on the quality of the dense reward model, the difficulty of the task, and the current training stage. The authors fix $k=3$ globally, which works well in these experiments but may not generalize without tuning.

One way to think about $k$: it controls the effective KL divergence between Agent B and Agent A. Small $k$ keeps this divergence small (similar distributions, limited diversity gain). Large $k$ grows this divergence (large distribution gap, harder to correct via importance sampling, potential reward hacking). The optimal $k=3$ is the sweet spot where Agent B’s distribution has moved just enough to be meaningfully different from Agent A’s without becoming adversarially different. Whether this sweet spot shifts for other tasks or model scales is an open empirical question.

Dense Reward Model Noise Robustness (Table 5)

A potential concern with RSPO is that it inherits all of the process reward model’s errors. If the PRM is inaccurate, Agent B’s exploration will be misdirected, and the trajectories it contributes to the replay buffer will be low-quality. The authors test robustness by adding Gaussian noise $\sigma$ to the dense reward signal during training:

Noise $\sigma$	SR (%)
0.0 (clean)	67.2
0.1	64.1
0.3	60.9

With $\sigma = 0.3$ added noise — a substantial perturbation given the reward scale of $(-1,1)$ — the success rate drops only 6.3 percentage points. The system does not collapse. The authors interpret this as evidence that the dense reward serves as an exploration guide rather than an optimization target: even a noisy guide points the agent toward useful regions of trajectory space, and since the final learning signal is the ground-truth outcome reward, the noise does not propagate into the learned policy.

This robustness result is practically important because it means RSPO can work with an imperfect off-the-shelf process reward model. You do not need a perfectly calibrated PRM to benefit from RSPO — a reasonable one suffices.

The degradation pattern — 67.2% → 64.1% → 60.9% as noise grows from 0 to 0.3 — is roughly linear, suggesting the PRM’s contribution to RSPO scales gracefully with its quality rather than exhibiting a cliff. This is the behavior you would expect if the PRM serves primarily as an exploration direction oracle (pointing toward useful regions of trajectory space) rather than as a precise per-step reward signal (whose exact values matter for computing advantages). The outcome reward re-labeling in Phase 2 means that the PRM’s values never directly appear in the final gradient computation — only in the behavioral influence on Agent B’s exploration paths. A noisier oracle explores in slightly worse directions but still explores, and the outcome reward provides the corrective signal downstream.

Replay Buffer Sampling Strategy (Table 2)

Sampling Strategy	ALFWorld SR (%)	WebShop SR (%)
Random	—	12.8
Variance-based	—	63.0
Reward-based	—	70.6

The gap between random and reward-based sampling (12.8% vs. 70.6%) is enormous — nearly a 6x factor. This is not a subtle hyperparameter choice; it is a core component of making RSPO work. Random sampling defeats the purpose of having a replay buffer: if the buffer contains mostly failed trajectories from Agent B, adding them to the on-policy batch just adds noise. Reward-based sampling ensures the off-policy data is dominated by Agent B’s successful explorations — precisely the cases where Agent B found trajectories that Agent A could not.

The variance-based strategy (63.0%) is interesting but underperforms. High-variance trajectories might represent hard tasks where the model is uncertain, but they are not necessarily more successful. Reward-based sampling directly selects for the signal RSPO is designed to propagate.

I find the 12.8% SR for random sampling particularly revealing. In that condition, RSPO is essentially adding random trajectories from Agent B’s distribution to the on-policy batch, re-labeled with outcome rewards. The fact that this dramatically hurts performance (baseline GRPO achieves 46.1% SR without any off-policy data) confirms that the off-policy data source matters enormously. Failed trajectories from Agent B, if added to the training batch, actively hurt training — they add examples of strategies that do not work, and the model presumably learns to partially imitate those bad strategies. Random sampling picks mostly failures because Agent B itself is not perfect; Agent B’s trajectory distribution is heavily skewed toward failures. Only with reward-based filtering does the off-policy data become net-positive.

This has a direct practical implication: if you implement RSPO without the reward-based sampling (e.g., using a naive deque-style replay buffer that stores trajectories in order), you will get worse performance than plain GRPO. The sampling strategy is not an implementation detail — it is part of the method’s core mechanism.

Key Analysis: Reward Hacking and Exploration

The Reward Hacking Trajectory

One of the most informative empirical results in the paper (Figure 3) shows what happens when you train exclusively on dense process rewards. The success rate on WebShop climbs steadily from roughly 0.25 to a peak near 0.65 around training step 80–100, then begins to fall. By step 140, the success rate has dropped back to around 0.35. Meanwhile, the dense reward metric $\hat{R}$ continues to rise monotonically throughout — the model is, by the metric it is optimizing, doing better and better.

This is the canonical reward hacking signature: the optimized metric diverges from the true objective. The model discovers sequences of actions that reliably trigger high scores from the process reward model but do not actually correspond to successful purchases. In WebShop’s case, this likely means the model learns to produce search queries and product selections that the PRM finds plausible but that do not satisfy the ground-truth product specification.

RSPO avoids this failure mode by construction: Agent B is only used for exploration, and its weights are never the final model. Agent A’s weights are only updated using outcome rewards, which are ground-truth and cannot be hacked. The temporal separation between exploration (dense rewards) and learning (outcome rewards) is what prevents the feedback loop from forming.

Trajectory Diversity Evidence (Table 6)

The paper measures state space diversity through identical state counts — the number of unique states that appear multiple times across trajectories at a given training step. A high identical state count means the policy is stuck in loops, repeatedly visiting the same states and emitting the same actions. A decreasing identical state count over training indicates the model is exploring new parts of the state space.

Method	Identical States (step 50)	Identical States (step 100)	Identical States (step 145)
GRPO	1011	1084	1167
RSPO	917	889	623

GRPO’s identical state count increases from 1011 to 1167 as training progresses. The model is becoming more deterministic and repetitive over time — converging to a narrow set of strategies. RSPO shows the opposite trend: starting from 917 and dropping to 623. As training progresses, RSPO’s agent visits more unique states and follows more diverse paths.

This is direct evidence that the reward-swap exploration mechanism does exactly what it is designed to do. Agent B’s dense-reward exploration genuinely expands the trajectory distribution, and the re-labeled trajectories teach Agent A to succeed via a wider variety of strategies rather than a single narrow path.

The practical implication is significant: a more diverse policy is more robust to environment variation. In ALFWorld, which involves 6 different task types with different object configurations, a policy that has learned diverse strategies handles novel configurations more gracefully than one that has memorized a single successful pattern.

Goodhart’s Law and the Role of Outcome Rewards

The reward hacking result and the exploration result together tell a unified story about Goodhart’s Law in LLM agent training. Goodhart’s Law, informally stated, says: “when a measure becomes a target, it ceases to be a good measure.” This is exactly what happens in dense-only training — the process reward, initially a useful proxy for the true objective, becomes the optimization target, and the agent finds ways to maximize the proxy that do not correspond to maximizing the true objective.

RSPO’s architecture is a structural defense against Goodhart’s Law. By maintaining a clear separation between the proxy (dense reward, used for exploration) and the true objective (outcome reward, used for learning), RSPO ensures that Goodhart’s dynamic can never fully activate. The proxy influences behavior (through Agent B’s exploration) but never directly optimizes the final policy. The true objective is always the ground truth for gradient computation.

This framing suggests a broader principle for RL fine-tuning of LLMs: whenever you use an approximate reward model, keep it out of the direct optimization loop if possible. Use it for data generation, filtering, or exploration guidance, but validate and update the policy only against ground-truth signals. RSPO is one concrete instantiation of this principle for the multi-turn agent setting.

Why GRPO Diverges and RSPO Converges

The contrast between GRPO’s increasing identical state count and RSPO’s decreasing one is more than a diversity metric — it reveals something about the long-term stability of each training algorithm. GRPO converges toward a deterministic local optimum: a small set of strategies that reliably succeed, surrounded by a large unexplored region of the strategy space. Once the model converges to this optimum, further training reinforces it rather than improving it — the identical state count increases because the model revisits the same states in every episode.

RSPO, by introducing fresh Agent B trajectories in each loop, repeatedly disturbs this equilibrium. The policy cannot fully converge to a narrow optimum because the off-policy data in each loop has a slightly different distribution, pushing the policy away from over-specialization. This is reminiscent of experience replay in deep Q-learning (where diverse historical transitions prevent catastrophic forgetting) but applied to the policy optimization objective rather than the value function update.

Limitations

Author-Stated Limitations

The authors identify three limitations in Appendix F. First, all experiments use models of at most 7B parameters. Whether the reward-swap mechanism provides similar benefits for 13B, 70B, or larger models is unknown. Larger models typically have stronger on-policy exploration through their greater representation capacity, which might reduce the benefit of Agent B’s diverse trajectories. Alternatively, the harder tasks tackled by larger models might require even more exploration. This is genuinely uncertain.

Second, the off-policy mixing ratio $\alpha$ is held constant throughout training. An adaptive schedule — increasing $\alpha$ early in training when on-policy data is low quality and decreasing it later when the policy is strong — might improve performance and reduce the sensitivity observed around $\alpha = 1/4$.

Third, RSPO’s performance inherits some dependence on the dense reward model quality. The noise robustness ablation shows this dependence is weak for moderate noise ($\sigma \leq 0.3$), but the authors acknowledge that a fundamentally broken PRM would likely cause problems.

Additional Limitations from Analysis

Beyond the author-stated limitations, several others are worth examining. The benchmark scope is narrow: both ALFWorld and WebShop are text-based environments with clean state transitions and binary/near-binary outcome rewards. Real-world agent tasks often involve stochastic environments, partial observability, and continuous or multi-dimensional reward signals. The paper does not test whether RSPO’s properties hold in these harder settings.

The compute cost of RSPO versus baselines is not discussed. RSPO requires training the dense reward model, running Agent B’s exploration trajectories, logging them to a replay buffer, and loading them back during Agent A’s training. In wall-clock time, this is likely 30–60% more expensive than plain GRPO or PPO for the same number of effective policy update steps. Whether the performance gains justify this extra compute is a practical question the paper does not address. A fair comparison would match total GPU-hours, not just training steps.

The off-policy correction via generalized clipping has a theoretical justification but no formal analysis of its convergence properties. Standard PPO has extensive convergence analysis; GRPO’s properties are less well-studied; generalized clipping adds another degree of freedom that the paper does not analyze theoretically. Practitioners applying RSPO to new domains would benefit from understanding when the correction is sufficient versus when the distribution gap is too large for any clipping-based correction.

The paper also does not evaluate RSPO’s effect on the model’s performance on held-out task types. Training on ALFWorld improves performance on ALFWorld, but does a model trained with RSPO generalize better to tasks outside its training distribution than a model trained with GRPO? This is an important question for practical deployment: a more diverse policy should, in theory, be more robust to distribution shift. This hypothesis is testable but not tested.

Finally, the paper does not analyze the failure cases of RSPO — the specific trajectories that RSPO generates successfully but that GRPO misses, and vice versa. A qualitative analysis of Agent B’s trajectories compared to Agent A’s would illuminate whether the exploration gain is coming from fundamentally different strategies (Agent B discovers a shortcut GRPO never found) or from the same strategies executed with higher frequency (Agent B revisits a strategy more often due to dense reward signal, filling the buffer with more positives). This distinction matters for understanding when RSPO will help and when it will not.

Critical Assessment: Weaknesses & Improvements

Missing Ablations

The paper provides four ablation studies but leaves several important variables untested. The most significant gap is the dense reward model architecture. The paper uses a specific combination (Llama-3B + MLP + tanh), but there is no ablation comparing this to alternatives: a smaller reward model, a reward model with the same architecture as the agent, or a rule-based process reward (e.g., step completion heuristics in ALFWorld). Without this ablation, it is impossible to know how much of RSPO’s gain comes from the reward-swap framework versus the specific PRM design.

Similarly, the paper ablates the noise robustness of the PRM but not the quality of the PRM’s base model. Llama-3B is used for both environments without justification. A smaller 1B or 500M base model might achieve similar exploration quality at lower compute cost. A 7B PRM might achieve better exploration quality and larger gains. This design space is entirely unexplored.

There is also no ablation on the choice of environment for Agent B’s exploration. Currently, Agent B explores in the same environment as Agent A. In principle, one could use a simpler or noisier version of the environment for exploration, which might reduce computational cost while preserving exploration diversity.

Evaluation Concerns

ALFWorld and WebShop are established benchmarks, but both have known limitations. ALFWorld’s 6 task types have limited structural diversity — the benchmark was designed for sample efficiency studies, not for evaluating generalization across diverse agent behaviors. The 88.8% GiGPO baseline already suggests that top-performing models are near ceiling on many task types; RSPO’s marginal gain on GiGPO (+1.6pp) might reflect ceiling effects rather than a fundamental improvement.

WebShop’s continuous score metric (0-1) and binary success rate can tell different stories about the same model. The paper reports both, but the discussion focuses on SR. Cases where RSPO improves SR but hurts Score (or vice versa) would reveal whether the method is improving genuine task completion or just optimizing for binary threshold-crossing behavior.

The paper does not report variance across training runs. Given the stochastic nature of RL training with multiple hyperparameter choices, a single run per configuration makes it difficult to distinguish genuine gains from lucky training trajectories. Standard practice in RL papers is to report mean and standard deviation over 3–5 seeds.

Theoretical Gaps

The generalized clipping mechanism has an intuitive motivation but no formal guarantees. The key question is: under what conditions is the importance sampling ratio $\pi_\theta / \pi_B$ well-behaved enough for gradient-based optimization? If $\pi_B$ assigns near-zero probability to some actions that $\pi_\theta$ assigns high probability to, the importance ratio can be extremely large, and clipping alone may not bound the gradient variance. The paper does not analyze the variance of the off-policy estimator or provide conditions under which the generalized clipping converges.

The reward-swap framework is motivated by intuition about exploration and exploitation but is not derived from a formal optimization objective. It is unclear whether RSPO is optimizing a well-defined objective or whether its good empirical performance is a consequence of implicit regularization through the multi-phase structure. Formalizing RSPO as solving a specific optimization problem would make it easier to extend, analyze, and compare to future methods.

Comparison to Missing Baselines

RSPO is compared to GRPO, GiGPO, PPO, and SPEAR. Several relevant baselines are absent:

Hindsight Experience Replay (HER): a classic off-policy RL technique that relabels failed trajectories with alternative goals they inadvertently achieved. HER is directly relevant to RSPO’s goal of expanding trajectory diversity and is used in goal-conditioned RL. The paper does not acknowledge this connection.

Self-play and data augmentation: methods where the agent generates synthetic training scenarios (e.g., by perturbing successful trajectories) to improve coverage. These achieve trajectory diversity through generation rather than exploration, and the comparison would clarify whether Agent B’s exploration adds value beyond what data augmentation could provide.

Curriculum learning: training on progressively harder tasks to avoid sparse reward problems. A curriculum that guarantees some early successes addresses the sparse reward problem from a different angle and might achieve similar diversity gains.

Reproducibility Concerns

The paper does not release code or a pretrained dense reward model checkpoint. Since RSPO requires a pre-trained Llama-3B-based PRM, practitioners attempting to reproduce the results must train this component independently. The paper provides the MSE training objective but not the training data, data preprocessing steps, or convergence criteria for the PRM.

The loop structure with phases interleaved makes implementation non-trivial. The interaction between the learning rate schedule, the loop boundary, and the replay buffer reset is not fully specified. For example: does the learning rate schedule reset at the start of each loop, or does it continue across loops? Does the KL reference model $\pi_{\text{ref}}$ update across loops or remain fixed to $\pi_{\text{base}}$?

The generalized clipping formula in Equation 4 requires computing $\pi_{\theta_{\text{old}}}(a|s) / \pi_B(a|s)$ for each off-policy sample. This requires storing the log-probabilities from Agent B at the time each trajectory was collected (since $\pi_B$ is no longer accessible by Phase 2 — the model has been updated back to Agent A’s weights). Storing per-token log-probabilities for every trajectory in the replay buffer doubles the buffer’s memory footprint compared to a standard GRPO implementation. The paper does not acknowledge this memory overhead, which may be significant for larger models or longer trajectories.

Additionally, the paper’s training setup uses Qwen2.5-1.5B and 7B with specific infrastructure (presumably multi-GPU distributed training given the model sizes and epoch counts). The training speed and batch size interact with the loop structure in ways that may affect reproducibility: if a different training setup changes the effective batch size, the replay buffer sampling ratio may need adjustment. None of this is spelled out.

Understated Design Choices

Several design choices in RSPO are presented as obvious but are actually load-bearing for the method’s success. The most important is the replay buffer reset after each loop ($D \leftarrow \emptyset$). This choice prevents stale trajectories from contaminating later training stages, but it also means Agent B must re-generate useful trajectories in every loop. If the loop count $k$ is small, Agent B may not produce enough high-quality trajectories to justify the overhead of the exploration phase. The paper treats the reset as self-evidently correct but provides no ablation comparing it to a persistent replay buffer with staleness weighting.

The choice of MSE loss for training the process reward model (Equation 7) is also underexamined. MSE loss treats all trajectory-reward pairs as equally informative, but trajectories near the success threshold (reward $\approx 0.5$) are likely noisier and less reliable than clearly successful (reward $\approx 1$) or clearly failed (reward $\approx 0$) trajectories. A weighted MSE or a classification-style loss with margin might produce a better-calibrated PRM that guides exploration more effectively.

The paper normalizes PRM output with $\tanh$ to constrain to $(-1, 1)$, but the MSE target $R_i$ is in $[0, 1]$. This means the PRM is being asked to approximate $[0, 1]$ targets with outputs from $(-1, 1)$ — the lower half of the $\tanh$ range is structurally unavailable to represent any valid training target. This is a minor implementation inconsistency that might slightly degrade PRM calibration, but it is never mentioned or analyzed.

Lack of Per-Task-Type Breakdown

The ALFWorld benchmark has six distinct task types (Pick, Look, Clean, Heat, Cool, Pick2) that differ substantially in difficulty and step length. Reporting only aggregate success rates obscures where RSPO’s gains are concentrated. If RSPO improves primarily on the hardest task types (Look, Pick2) and is neutral or negative on easy ones, that would be strong evidence for the exploration-diversity mechanism. If RSPO improves uniformly across all task types, the mechanism might be acting differently than hypothesized — for example, improving the model’s language-following or command-formatting rather than its planning strategy. This breakdown is missing from Table 1, which I consider a significant omission for a paper that makes specific claims about exploration diversity.

Concrete Improvement Suggestions

Adaptive $\alpha$ scheduling: The cliff in performance at $\alpha = 1/4$ suggests the benefit of off-policy data is highly context-dependent. A simple improvement would be to schedule $\alpha$ based on the current policy’s on-policy success rate: use higher $\alpha$ when the on-policy success rate is very low (the model needs diverse trajectories most) and lower $\alpha$ when the on-policy success rate is already high (diverse off-policy data adds noise rather than signal).

Loop count $k$ scheduling: Similar to $\alpha$, $k$ could adapt based on detected reward hacking signals. If the dense reward $\hat{R}$ for Agent B’s trajectories is rising while the re-labeled outcome reward is falling, $k$ should decrease. A simple heuristic detector for this divergence could prevent the performance degradation at $k=5, 10$.

Persistent PRM with online updates: The current PRM is trained offline on fixed data. An online variant that updates the PRM on Agent A’s current trajectories would allow the reward model to improve as the policy improves, potentially better guiding exploration in the later stages of training when the easy trajectories have already been learned.

Theoretical analysis of generalized clipping: Deriving a bound on the variance of the off-policy estimator as a function of the KL divergence between $\pi_B$ and $\pi_{\theta_{\text{old}}}$ would provide practitioners with a principled criterion for when the generalized clipping correction is sufficient versus when $\pi_B$ has drifted too far from $\pi_\theta$ for importance sampling to be reliable.

Evaluation on longer horizon tasks: Testing RSPO on tasks requiring 50+ steps (e.g., complex coding tasks, multi-document summarization with retrieval, long-horizon planning in complex environments) would reveal whether the exploration benefits scale with task difficulty or are limited to the moderate-horizon settings tested here.

Comparison to alternative exploration strategies: Methods like epsilon-greedy with a temperature schedule, nucleus sampling with adaptive temperature, or population-based training (running multiple independent agent copies and sharing trajectories across them) address the exploration problem from different angles. A direct comparison would help isolate what is unique about RSPO’s cyclic dense-reward exploration versus these simpler alternatives. It is possible that a much simpler approach — increasing the sampling temperature during collection episodes — achieves comparable trajectory diversity without requiring a PRM at all.

Multi-task and transfer evaluation: Training on both ALFWorld and WebShop simultaneously, or sequentially, would reveal whether RSPO’s trajectory diversity helps with multi-task generalization or if the exploration benefit is task-specific. This would be a stronger test of the claim that diverse trajectories produce a more robust policy.

Conclusion

RSPO is a practically motivated and empirically effective framework for multi-turn LLM agent training. Its core insight — using dense rewards for exploration while training on outcome rewards for correctness — resolves the fundamental tension between signal density and signal accuracy without introducing new hyperparameter complexity beyond what GRPO and PPO already require.

I find the paper most compelling in its empirical evidence for the exploration problem (Table 6’s identical state counts) and the reward hacking timeline (Figure 3). These two results together make the strongest case for RSPO’s design: the diversity problem is real (GRPO gets stuck in loops), the naive fix (dense-only training) causes reward hacking, and the reward-swap structure avoids both. The generalized clipping mechanism is a technically correct and non-obvious contribution that addresses a real implementation pitfall.

The paper’s weaknesses are in breadth rather than in the core claims. The evaluation covers two text-based environments with similar task structures; the theoretical analysis of the off-policy correction is informal; and several important ablations (PRM architecture, per-task breakdown, seed variance) are absent. These are the standard limitations of a well-executed but narrowly scoped systems paper. They do not undermine the core result, but they do leave open the question of whether RSPO’s gains generalize to harder settings.

My overall assessment: the paper is a solid empirical contribution with a clear, well-motivated core idea and honest reporting of its scope and limitations. It is worth reading for anyone working on multi-turn LLM agent RL training, and RSPO is worth implementing if you have a process reward model available and are finding that GRPO’s exploration is the bottleneck. The method is robust to the practical imperfections you will encounter (noisy PRM, $\alpha$ in the right range), and the sampling strategy insight (reward-based over random or variance-based) is a standalone result that should inform any off-policy LLM training pipeline.

The main open questions are theoretical (convergence properties of generalized clipping, formal objective characterization of the reward-swap framework) and empirical (performance on models larger than 7B, performance on longer-horizon tasks, robustness to PRM architecture choice, variance across training seeds, per-task-type breakdown on ALFWorld). For practitioners working with small-to-medium LLM agents on text-based environments, RSPO offers a low-risk improvement that wraps existing training pipelines with minimal additional infrastructure. The observation that RSPO+PPO achieves 69.3% SR on WebShop compared to PPO’s 54.4% — a 14.9 percentage point gain — with the same base model and objective family is the most actionable result: if you are already training agents with PPO, adding the reward-swap loop is likely to be worth the effort.

Broader Implications for LLM Agent Training

The RSPO paper is part of a broader shift in LLM training thinking: from the view that RL is a fine-tuning step applied after SFT to the view that RL is a core training modality that requires its own architectural engineering. Methods like GRPO, GiGPO, and RSPO are developing the infrastructure for that shift — better advantage estimation, better off-policy handling, better exploration mechanisms.

What strikes me about RSPO is that it solves the exploration problem by re-purposing the very failure mode it is trying to avoid. Dense reward training leads to reward hacking — but if you apply it briefly and then discard the weights, you harvest the exploration benefit of dense rewards without suffering the hacking consequences. This is an elegant reframing of a known problem: instead of “how do we prevent dense rewards from causing reward hacking,” RSPO asks “how do we use dense rewards safely, for just long enough to help.” That reframing is, I think, the paper’s most durable intellectual contribution, independent of the specific implementation details.

Looking forward, the most interesting direction is whether the reward-swap principle generalizes beyond text agents. In robotic control, where outcome rewards are similarly sparse (task success) and dense rewards can be engineered from motion quality metrics, a similar cyclic exploration framework might provide analogous diversity gains. The separation between “use dense rewards for exploration” and “use outcome rewards for learning” is a design principle that does not depend on the LLM modality — it is a general statement about how to decouple the exploration and exploitation functions of reward signals. I expect this principle to appear in future work under different names and with different implementations, in domains ranging from tool-use agents to code generation to scientific hypothesis search.

Practical Takeaways

For practitioners considering RSPO: the method is most likely to help when (1) your base RL algorithm’s success rate is in the 40–70% range — low enough that exploration is the bottleneck, high enough that some successful trajectories exist to anchor the dense reward model; (2) you have access to a reasonable process reward model (or can train one with modest data); and (3) your task involves long trajectories where sparse rewards create significant credit assignment problems.

RSPO is unlikely to help if your task already has moderate step-level feedback from the environment (e.g., intermediate completion bonuses), if the success rate is extremely low (below ~20%, where Agent B cannot generate enough successful trajectories for reward-based sampling to find), or if the action space is small enough that on-policy exploration is already sufficient to cover the relevant strategies.

The method requires no changes to your base training loop beyond wrapping it in the cyclic structure, implementing the generalized clipping correction, and maintaining the replay buffer with reward-based sampling. This relatively low implementation overhead — given that you already need a PRM — makes it a pragmatic addition to any multi-turn agent training pipeline that is struggling with the exploration-convergence balance.

The one implementation caveat worth repeating: do not forget to store Agent B’s per-token log-probabilities at trajectory collection time. Without them, the generalized clipping correction cannot be computed, and you will fall back to standard clipping with incorrect importance weights — an error that will silently produce suboptimal performance with no obvious diagnostic signal.

If the paper releases code, the implementation quality of the replay buffer (especially the reward-based sampling and the log-probability storage) will be the first thing I check. Those two details are where most off-policy LLM training implementations go wrong in practice, and getting them right is what separates a working RSPO implementation from a plausible-looking one that quietly underperforms.

The paper’s arXiv identifier is 2607.04713. It was posted July 6, 2026. Follow the authors — Qiang Liu, Taian Guo, Ruizhi Qiao, and Xing Sun — for follow-up work extending RSPO to larger models or more complex agent environments.

Appendix: Implementation Checklist

For practitioners who want to implement RSPO from the paper description, here is a concise checklist of the non-obvious implementation details that are easy to get wrong:

Replay buffer setup

• Store complete trajectories, not individual $(s, a, r)$ transitions
• Each trajectory entry must include: token sequence, per-step dense rewards, scalar outcome reward $R(\tau)$, and per-token log-probabilities $\log \pi_B(a_t|s_t)$ at collection time
• Sampling is proportional to $R(\tau)$, not uniform — implement a weighted reservoir sampler or a sorted priority queue

Agent B training

• Agent B uses the standard on-policy objective (GRPO or PPO) with dense rewards $\hat{r}_t$ from the PRM
• Agent B’s policy parameters are updated every loop iteration $k$; they are not frozen
• The PRM is trained once before the outer loop begins and is not updated during RSPO training

Generalized clipping

• The importance weight is $\rho_t = \pi_\theta(a_t|s_t) / \pi_B(a_t|s_t)$, not $\pi_\theta / \pi_{\theta_{old}}$
• The clipping bounds shift to center around $\pi_B / \pi_{\theta_{old}}$ rather than 1.0 — this requires storing $\pi_{\theta_{old}}$ per trajectory as well
• If you forget to store $\log \pi_B$ at collection time, you cannot recover them later without re-running Agent B, which is expensive

Mixed objective

• $J_{Total} = (1-\alpha) J_{On} + \alpha J_{Off}$ with $\alpha = 1/8$; the paper is explicit about this ratio
• $J_{On}$ is computed on fresh Agent A trajectories; $J_{Off}$ is computed on replayed Agent B trajectories
• Both gradients are accumulated in the same backward pass; do not alternate between them

Loop hyperparameters

• Outer loop: 150 total training iterations
• Inner Agent B loop: $k=3$ consecutive Agent B updates before switching back to Agent A
• Replay buffer capacity: not specified explicitly in the paper — start with 512 trajectories and tune based on memory

Verification steps

• After training Agent B for $k$ steps, verify the per-step reward distribution has not collapsed (a sign of reward hacking)
• Monitor Agent A’s success rate separately from the mixed objective loss — the off-policy gradient can temporarily depress the loss without improving success rate
• Check trajectory diversity in the replay buffer using the edit-distance metric from Table 6; a diversity score below 0.3 suggests the buffer is too homogeneous