AgentRewind: Giving Long-Horizon LLM Agents an Undo Button That Actually Undoes Things

Review date: 2026-08-17 Author: Zhongzhu Zhou Paper reviewed: AgentRewind: Recoverable Execution for Long-Horizon LLM Agents Paper authors: Yu Zhuang, Kefei Chen, Yitong Duan, Shuxin Zheng, Jian Li, Xu-Yao Zhang (University of Chinese Academy of Sciences; IIIS, Tsinghua University; Zhongguancun Academy; Institute of Automation, CAS) arXiv: 2608.14380 Venue/Status: arXiv preprint, August 2026

1. The problem: agents can only ever go forward

If you’ve watched an LLM agent work through a long software-engineering task — say, migrating a service, fixing a multi-file bug, or cleaning up a repository — you’ve probably seen this failure pattern: the agent makes a plausible-looking decision early on, that decision turns out to be subtly wrong, and every subsequent step it takes is now built on top of a broken foundation. The agent doesn’t “know” it made a mistake five steps ago; all it can do is keep appending new actions to a trajectory that is already compromised. Sometimes it patches around the problem. Often it can’t, because the damage isn’t just in its own reasoning context — it’s already been written to disk. A file got overwritten. A git history got rewritten. A configuration got silently corrupted. By the time the agent (or its human overseer) notices, the “fix” isn’t a next action, it’s a full teardown-and-rebuild.

This paper’s core observation is structural, not anecdotal: standard agent execution loops are forward-only. At every step, an LLM produces a decision, the environment transitions, and the agent’s context is updated by appending the new observation. There is no mechanism, built into the loop itself, for returning to an earlier point. Existing mitigations — better planning up front, safety monitors that block risky actions during execution — reduce the probability of an error at each step, but they do nothing once an error has already happened and propagated. And because probability of failure per step, however small, compounds over a long trajectory, long-horizon tasks are exactly where this gap bites hardest.

AgentRewind’s proposal is refreshingly literal: give the agent an actual rewind button — one that restores both its conversational context and the external environment (the filesystem it’s been modifying) to a jointly consistent earlier snapshot, and that carries forward a compact memory of what went wrong so the agent doesn’t just repeat the same mistake. The paper backs this with a genuinely new benchmark, MettleBench, built specifically to stress long-horizon, multi-requirement engineering tasks rather than the single-objective tasks most agent benchmarks use, and shows consistent, sometimes dramatic (+25.6 points task success in the best configuration) gains across seven base models, four execution strategies, and three different agent harnesses.

This review works through the mechanics of AgentRewind step by step — the checkpointing scheme, the rewind decision process, and exactly what does and doesn’t get undone — then unpacks MettleBench’s design, walks through the full experimental picture including the ablations, and closes with a critical look at what the paper leaves underexplored.

2. Prerequisites: what you need going in

The agent-environment interaction loop. A “tool-using” or “agentic” LLM setup, as opposed to a single-turn chatbot, works in a loop: the model receives a task instruction and an initial environment state, generates a decision (usually a tool call — run a shell command, edit a file, query an API), the environment executes that action and returns an observation, and that observation gets appended to the model’s context for the next step. This continues until the model decides the task is done or some stopping condition (step limit, repeated-failure detector) is hit. Frameworks like mini-SWE-agent, Qwen-Agent’s FnCallAgent, and smolagents’ CodeAgent are all concrete implementations of this loop, differing mainly in how they expose tools to the model (JSON function-calling vs. having the model literally write Python) and how they structure the surrounding control flow.

Why “long-horizon” is qualitatively different from “long.” A task isn’t long-horizon just because it takes many steps — a repetitive data-labeling task with 500 near-identical steps is long but not really long-horizon in the sense this paper cares about. What makes a task long-horizon and hard to recover from is dependency structure: later steps depend on earlier steps having produced correct intermediate state, and undoing a mistake requires more than “try again” because the mistake has already changed the world the agent is operating in. This is precisely the distinction the paper’s benchmark, MettleBench, is built to capture (more on this in §4).

Checkpointing and rollback, as a general systems idea. The core mechanism here — periodically snapshot state, and provide a way to restore an earlier snapshot when something goes wrong — is one of the oldest ideas in systems design, used in database transactions, VM live-migration, and process-level checkpoint/restart tools like CRIU (Checkpoint/Restore In Userspace) and DMTCP. What’s specific to the agent setting is that there are two pieces of state that must be checkpointed together and kept aligned: the LLM’s conversational context (which encodes what the agent “believes” and “remembers”) and the external environment (the actual filesystem/workspace it has been mutating). Get these out of sync — restore one without the other — and you get an agent that either “remembers” doing things it can no longer see evidence of, or sees a clean filesystem but still “remembers” the mess it made, neither of which is coherent. This joint-consistency requirement is the crux of why this is a genuinely different engineering problem from either “rewind a git repo” or “roll back a conversation” in isolation.

Rewind vs. retry vs. restart — three different recovery philosophies. It’s worth being precise about vocabulary before diving into the mechanism, because the paper’s baselines correspond to distinct philosophies that are easy to conflate:

• Retry / Continue: keep the same context and environment, just try a different next action. This is what most agents do by default when they receive negative feedback — nothing is undone, the mistake and its consequences are still present, the agent just tries to work around them.
• Restart: throw away both context and environment, go back to the pristine initial state, and start over — optionally carrying forward some distilled “lessons learned” as text.
• Rewind (this paper’s contribution): go back to an intermediate point that is neither “everything” nor “nothing” — retaining validated progress made before a specific mistake while discarding only the mistake and its downstream consequences.

Rewind is strictly more expressive than the other two: full restart is rewind-to-the-very-beginning, and continue is (trivially) rewind-to-the-present. The interesting engineering content is in making arbitrary intermediate rewind targets both available (there’s a menu of checkpoints to choose from) and safe (restoring one doesn’t leave dangling inconsistent state).

3. AgentRewind’s architecture: three moving parts

Figure 1 (reproduced below) gives the full picture. AgentRewind sits as a transparent runtime layer between the agent and the environment — it doesn’t replace the agent’s decision-making, it instruments the interaction channel.

(a) During normal, forward execution: two recorders operate continuously and transparently.

• The Agent Context Recorder logs every LLM input/output pair — the messages sent to the model and the response (including any tool calls) it produced.
• The Environment State Recorder logs every tool-induced change to the controlled environment — concretely, file-level diffs to the workspace directory tree.

At every LLM decision boundary $t$, these two streams are combined into a state checkpoint $d_t = (c_t, s_t)$ — the agent’s context $c_t$ paired with the environment state $s_t$ at that exact moment. This pairing is the whole trick: because both halves are captured at the same logical instant, restoring $d_t$ later guarantees the agent’s beliefs and the world it’s looking at are mutually consistent, which is not automatically true if you checkpoint conversation history and filesystem state on separate, unsynchronized schedules.

(b) When the agent decides forward progress has stalled, it can trigger rewind, which is exposed to it as two additional tools alongside its normal shell/tool interface:

1. backtrack_candidates(reason) — returns a list of eligible earlier checkpoints (up to 80 candidates are shown), along with checkpoint metadata $\eta_t$ summarizing what happened in the execution segment between $d_t$ and $d_{t+1}$ — essentially a compressed diff-plus-narrative for each candidate segment, so the agent can decide which point in its own history is worth returning to without having to re-read the entire raw trajectory.
2. backtrack_commit(record_uid, memory_summary, reason) — commits to a specific target checkpoint $d_k$ and supplies a rewind memory $m$, a free-text note the agent writes to its future (restored) self, summarizing what it learned from the discarded branch: which hypothesis turned out to be false, which approach didn’t work and why, what to try instead.

4. The formal mechanics, unpacked

The paper gives a compact formalization that’s worth deriving carefully rather than skimming, because the notation is doing real conceptual work.

Standard forward execution. At step $t$, given context $c_t$, the policy (the LLM) samples a decision:

$$u_t \sim \pi(c_t)$$

The environment transition function $T$ takes the current environment state $s_t$ and the decision $u_t$, and produces a new state and an observation:

$$(s_{t+1}, o_{t+1}) = T(s_t, u_t)$$

The context is then updated by an update function $U$ that appends the new decision and observation to the running history:

$$c_{t+1} = U(c_t, u_t, o_{t+1})$$

Chained together, this produces a forward-growing trajectory:

$$\tau = \big( (c_0, s_0), u_0, o_1, (c_1, s_1), \ldots, (c_T, s_T) \big)$$

The key structural fact the paper is setting up here: under this formalization, $\tau$ can only ever be extended, never truncated. If $u_k$ (some early decision) was wrong, its effects are baked into every $c_{k+1}, c_{k+2}, \ldots$ and every $s_{k+1}, s_{k+2}, \ldots$ that follows, by construction — the update function $U$ and transition function $T$ have no “undo” direction. Even if the agent later recognizes the error (which LLMs are often capable of doing, in isolation — self-critique is not the bottleneck here), all it can do within this formalism is append corrective actions on top of an already-corrupted state.

What a checkpoint is, formally. AgentRewind’s checkpoint is simply the pairing already described:

$$d_t = (c_t, s_t)$$

recorded at every decision boundary, plus checkpoint metadata $\eta_t$ describing the segment between $d_t$ and $d_{t+1}$ (used purely for presenting candidates to the agent — it’s a UI/decision-support artifact, not part of the state that gets restored).

What happens when a rewind is committed to target $k$. This is the two-line mechanism at the heart of the paper:

$$s_k' \leftarrow s_k$$ $$c_k' \leftarrow \text{Inject}(c_k, M)$$

Read this carefully: the environment is restored exactly to its checkpointed state $s_k$ — no modification. The context, however, is not restored verbatim; it’s restored and then augmented by injecting the accumulated set of rewind memories $M$ (the union of every rewind-memory note written across every rewind this run has performed so far, per Eq. 4: $M \leftarrow M \cup \{m\}$). This asymmetry is deliberate and, I think, the single most important design decision in the whole paper: the environment must be pristine (any leftover trace of the failed branch would defeat the purpose of rewinding), but the agent’s knowledge that the branch failed, and why, is exactly the value being carried forward. Losing that would make rewind indistinguishable from a blind restart to that point.

Execution then resumes from this restored pair, producing a genuinely new continuation:

$$\tau' = \big( (c_k', s_k'), u_k', o_{k+1}', \ldots \big)$$

Notice this is not the same $u_k$ replayed — the agent, now equipped with the rewind memory, is free to (and in the FITS case study below, does) make a different decision at the same decision point.

5. Algorithm 1: the rewind decision loop, pseudocode

The paper doesn’t present this as literal numbered pseudocode, but the mechanism described across the “Rewind Module” and Appendix C sections decomposes cleanly into one. I reconstruct it here because unpacking it step-by-step clarifies several subtleties (particularly around what triggers a rewind and what “restore” actually does under the hood) that are easy to miss reading prose alone.

Algorithm 1: AgentRewind Runtime Recovery Loop
Input: task instruction x, initial state (c_0, s_0), rewind budget = unlimited
State: trajectory log L = [], rewind-memory set M = {}

 1  t <- 0; L.append(record(kind=llm/tool, ...))   # transparent recording begins immediately
 2  loop:
 3      u_t <- LLM_decision(c_t)                    # normal forward step
 4      if u_t is a tool call (not a rewind tool):
 5          (s_{t+1}, o_{t+1}) <- Environment.execute(s_t, u_t)
 6          record file-level diffs of s_t -> s_{t+1} as part of checkpoint d_t's metadata
 7          c_{t+1} <- U(c_t, u_t, o_{t+1})
 8          d_{t+1} <- (c_{t+1}, s_{t+1})            # new aligned checkpoint recorded
 9          t <- t + 1
10          if repeated-failure condition met (same first-unsatisfied criterion, 5x in a row):
11              TERMINATE run as unsuccessful
12      elif u_t == backtrack_candidates(reason):
13          candidates <- top-80 checkpoint metadata entries eta_j from L, j <= t
14          o_{t+1} <- present candidates to agent (does NOT count as an environment step)
15          c_{t+1} <- U(c_t, u_t, o_{t+1}); t <- t + 1
16      elif u_t == backtrack_commit(record_uid=k, memory_summary=m, reason):
17          M <- M union {m}                        # Eq. 4: accumulate, never overwrite
18          s'_k <- s_k                              # Eq. 5a: environment restored verbatim
19          c'_k <- Inject(c_k, M)                   # Eq. 5b: context restored + memory injected
20          discard everything in L recorded strictly after d_k (the "degraded suffix")
21          (c_t, s_t) <- (c'_k, s'_k); t <- k        # resume the run FROM checkpoint k
22          continue loop                              # agent now generates a NEW suffix
23  until task success (Eq. 8: all criteria satisfied) or terminated at line 11

A few things worth calling out explicitly:

• Line 4-11 is exactly the ordinary agent loop — rewind is not a replacement for normal execution, it’s an additional control action available at every step, alongside whatever tools the agent already has (line 12 and 16 branches). This is important for the harness-adaptation story (§9 below): you don’t need to redesign an agent’s control flow to add AgentRewind, you need to add two more items to its tool menu.
• Line 14: listing candidates costs a step but not an environment mutation. The agent “spends” a turn calling backtrack_candidates, but nothing in the world changes — it’s pure information-gathering, and the paper’s trace-length accounting (used throughout Table 1 and Figure 3) counts it as one more LLM/tool event, same as any other action.
• Line 20: “discard the degraded suffix” is not literal deletion. Per Appendix C.2/C.3, rewind actually forks — the trajectory log after a rewind is not physically erased; a new file is opened whose header records which run it forked from and which node it forked at. This matters for auditability (you can inspect exactly what happened on the abandoned branch) without it being part of the live context or environment going forward.
• Line 21: resuming from $k$, not $t$. This is the entire point — the agent’s step counter, in effect, jumps backward, and everything from $k+1$ to the old $t$ is simply gone from the active trajectory (though preserved in the log per the previous point).

6. What actually gets restored, and — critically — what doesn’t

This section is where the paper is most honest, and it matters a lot for judging where AgentRewind will and won’t help in practice.

The environment recovery boundary is the workspace filesystem tree, and nothing else. Per the paper’s own “External Environment Recovery Boundary” subsection: at each checkpoint, AgentRewind records file-level changes (which files were modified, created, or deleted, tracked via commit hashes — Appendix C.4 shows the filesystem block in each record carries before_commit/after_commit hashes and a diff_summary). Rewinding reverts later modifications, restores deleted files, and removes newly-created files — a fairly complete story for the filesystem.

But: “Effects outside the workspace filesystem, such as network requests, external-service calls, and external runtime state, cannot be undone.” If the agent’s mistaken branch sent an email, made an API call that charged money, pushed a commit to a remote (not local) repository, or mutated a database it doesn’t have exclusive local control over, rewinding the local workspace does nothing to that side effect. The paper is careful to note that because the retained prefix is restored from the execution log rather than re-executed, those external effects at least aren’t triggered again on rewind — but the ones that already happened during the discarded suffix stay happened. This is a real, structural limitation, not a minor caveat, and I come back to it in the critical-analysis section.

How the mechanism attaches to different agent harnesses without modifying them (Appendix C.1). This is a nice piece of engineering worth walking through because it explains why the paper can credibly claim “harness-agnostic” rather than just asserting it. The layer attaches at exactly two points — the LLM call and the tool call — corresponding to the context recorder and environment-state recorder. For frameworks that route LLM calls through a standard client (OpenAI SDK, LiteLLM), AgentRewind auto-instruments by replacing the client’s completion entry point, forcing non-streaming responses so nothing is missed mid-stream. For frameworks that don’t (or that use LangChain/LangGraph), a per-harness wrapper intercepts calls directly. Tool calls are captured by wrapping the tool-execution function itself, regardless of what specific tools a harness exposes.

Two harness-specific wrinkles the appendix flags honestly: (1) presenting the two rewind tools requires speaking each framework’s own tool-definition dialect — mini-SWE-agent and FnCallAgent both use ordinary function-calling schemas, but CodeAgent (smolagents) expects the model to write Python, so there the rewind “tool” has to be exposed as an ordinary Python callable instead of a JSON schema; (2) signaling a committed rewind out of the agent’s control loop is harness-specific plumbing — for CodeAgent specifically, its executor catches ordinary Python exceptions and returns them to the agent as observations (i.e., it treats them as recoverable errors the agent should see, not as a runtime signal), so the rewind commit has to be raised as a subclass of BaseException to escape that catch and actually reach the runner. The authors report this adaptation work comes to roughly 150-300 lines of code per harness — small, but not zero, and a real tax on adopting this for a new/custom agent framework.

7. Deadlock-safety and correctness: is it actually safe to touch live state mid-decode?

Unlike some of the KV-cache/memory-management systems papers in this same broader literature that need formal deadlock arguments because they’re manipulating GPU memory concurrently with active compute kernels, AgentRewind’s safety story is comparatively simpler because rewind is a synchronous, blocking operation from the agent’s point of view — but it’s still worth being precise about why it’s safe, because “restoring a workspace mid-execution” sounds hairier than it is.

The mechanism (Appendix C.3): the backtrack_commit tool doesn’t return a normal tool result. It raises an exception, which propagates out of the agent’s session and is caught by the runner — the outer harness-specific process that launches the agent on a task and collects its result, as distinct from the agent’s own control loop. The runner then writes down everything the next attempt needs (which node to go back to, what memory to inject) and starts a fresh process with the workspace restored to the target commit and the context reconstructed from the log. Because the restore happens in a freshly-started process operating on a workspace whose earlier state is deterministically reconstructible from tracked file-level diffs (not from re-running anything), there’s no window where two processes are concurrently mutating the same files, and no possibility of a rewind racing against an in-flight tool call — the old process is simply gone by the time the new one starts. This sidesteps the concurrent-access hazards that would exist in a design where rewind tried to mutate live state underneath a still-running agent loop.

Design choice worth flagging: why exception-raising rather than a normal return value? The obvious alternative is to have backtrack_commit return a normal tool result (e.g., {"status": "rewound", ...}) and let the harness’s ordinary control flow handle the transition. The paper doesn’t spell out the rationale, but the exception-based design has a clear advantage: it guarantees the current agent turn cannot continue past the commit point under any circumstances, regardless of what the surrounding harness code does with a returned value — an ordinary return could in principle be ignored or mishandled by harness-specific control flow that wasn’t written with rewind in mind, silently corrupting the semantics. The cost, as the CodeAgent case shows, is that some harnesses’ exception-handling machinery actively fights this design and has to be specifically routed around.

8. Design-choice discussion: three more decisions worth interrogating

Why is rewind exposed as an agent-invoked tool, rather than an automatic trigger (e.g., after N consecutive failures)? The paper’s design puts the decision to rewind entirely in the agent’s hands — it decides, based on validation feedback and its own assessment, when “further progress along the current trajectory is unlikely” (per Figure 1(b)‘s framing). The alternative — an automatic policy, e.g., “rewind after 3 consecutive failures at the same criterion” — would remove a layer of LLM judgment that could go wrong, but it would also be a much blunter instrument: it can’t distinguish “this specific branch is unrecoverable” from “this branch just needs one more attempt,” and it can’t choose a good target checkpoint (as opposed to always rewinding one step). The paper doesn’t run a controlled ablation of agent-decided vs. automatic rewind, which is a real gap — we don’t actually know how much of AgentRewind’s gain depends on the model’s own judgment about when to rewind being decent, versus the mechanism itself. Given that “Safety Review” (an automated external judgment mechanism for a different decision — blocking unsafe actions) performs markedly worse than plain Continue under GPT-5.4 in this same paper (Table 13: 34.1% vs 62.2% success), there’s at least a suggestive worry that automated intervention mechanisms in this pipeline can backfire when the automated judge’s calibration doesn’t match the task distribution well; whether that generalizes to an automated rewind trigger is untested.

Why cumulative rewind memory rather than replacing the memory at each rewind? Per Eq. 4, $M$ only grows ($M \leftarrow M \cup \{m\}$) — every rewind’s memory note is retained and injected together at every subsequent rewind, never discarded even if a later rewind supersedes an earlier lesson. The obvious alternative is to keep only the most recent note, or to have the agent explicitly revise/merge notes. The ablation in Table 5 shows removing rewind memory entirely costs 36.6 points of success rate (87.8% → 51.2%) and 24.9 points of checklist progress (94.3% → 69.4%) — by far the largest single-component effect reported alongside environment rewind — so some form of memory clearly matters enormously. But cumulative-and-never-pruned is a specific choice with a specific failure mode: on a task requiring many rewinds, the injected memory block could grow to dominate the context window, diluting attention on the actual task instruction, and the paper’s own Appendix B.3 admits Restart-with-Experiences’ analogous accumulating experience file “reaches a few hundred lines for the tasks that restart most often” — the same growth dynamic plausibly applies to AgentRewind’s memory, though the paper doesn’t report memory-block length statistics for AgentRewind specifically, which would have been useful to include.

Why 80 checkpoint candidates shown, and why is that number fixed across every configuration in Tables 9-12? The paper reports “checkpoint candidates shown: 80” as a fixed setting everywhere rewind is used, but never justifies the number or reports sensitivity to it. This is a meaningful design surface: too few candidates and the agent may not have visibility into the actually useful rewind target (especially on MettleBench’s longer traces — recall average Continue trace lengths range up to 337 events for some models, so 80 candidates is already a substantial compression, not “show everything”); too many, and the agent has to spend more of its own context budget parsing the candidate list rather than reasoning about the task. No ablation over this parameter is reported, which is a missed opportunity given how directly it should interact with the “medium/long-horizon tasks benefit more” finding of Figure 4 — if longer-horizon tasks have proportionally more candidates competing for the same fixed 80-slot budget, the visibility-into-useful-targets problem should get worse, not better, as horizon grows, yet the empirical pattern is the opposite. That tension is worth understanding and the paper doesn’t address it.

9. MettleBench: why a new benchmark, and how it’s built to be hard in the right way

The paper’s argument for needing a new benchmark is precise and, I think, correct: most existing agent benchmarks (SWE-bench, WebShop, ALFWorld) frame each task around one primary requirement, even when solving it takes many steps. That’s a problem for evaluating recovery mechanisms specifically, because a task with one requirement gives you a binary success/fail signal and nothing about how far a failed run actually got, and — more importantly for this paper’s thesis — single-requirement tasks don’t naturally create the interdependency structure where an early mistake becomes structurally hard to undo without discarding good later work too.

Formal task structure. A MettleBench task is:

$$\mathcal{T} = (x, s_0, \mathcal{U}, G), \qquad G = (g_1, \ldots, g_n)$$

where $x$ is the natural-language instruction, $s_0$ the initial environment state, $\mathcal{U}$ the agent’s decision space, and $G$ an ordered list of $n$ binary acceptance criteria $g_i: \mathcal{S} \to \{0,1\}$. Task success requires all criteria simultaneously satisfied at the final state:

$$\text{Succ}(s_T) = \bigwedge_{i=1}^n \big( g_i(s_T) = 1 \big)$$

Crucially, the ordering is not arbitrary — criteria are interdependent through shared environment state: satisfying one can enable or block a later one, and a later action can silently regress an earlier satisfied criterion. To measure partial credit, the paper defines checklist prefix progress: let $\ell$ be the length of the longest prefix $(g_1, \ldots, g_\ell)$ that is fully satisfied in order (i.e., $g_i(s_T)=1$ for all $i \le \ell$):

$$\rho(s_T) = \frac{\ell}{n}$$

This metric is doing something subtly different from “fraction of criteria satisfied” — it specifically rewards ordered progress and penalizes a run that satisfies, say, criteria 1, 2, and 5 but not 3 or 4 (that run gets $\rho = 2/n$, not $3/n$), which matches the feedback mechanism the benchmark’s backend actually gives the agent: on every submission, it reveals only the natural-language feedback for the first unsatisfied checklist item, never the full checklist, and never criteria beyond the current blocking one. This is a deliberately narrow feedback channel, modeling how real-world CI/review feedback usually works (you find out about the next blocking issue, not a full report card).

Composition and construction pipeline. 82 tasks, drawn from five existing benchmarks (Terminal-Bench 2.0, ProgramBench, SWE-bench, ProjectEval, GitTaskBench — see the reproduced Figure 2 pie chart), with the underlying engineering artifacts and executable environments retained, but task instructions rewritten to bundle multiple interdependent requirements into a single assignment. Two structural properties are what actually create the “hard to undo” character, and the paper is explicit about them rather than leaving them implicit: (1) all criteria act on the same environment state, so satisfying one can hinder another; (2) many real engineering operations are genuinely destructive at the tool level — rewriting git history discards unreferenced objects, dropping a database column discards its values, regenerating a derived file overwrites its source — so an agent that does things in a bad order can create damage that is not just “wrong” but literally unrecoverable by further forward action, only by having kept an earlier, uncorrupted state around.

The security-incident worked example (Appendix A.7) is genuinely illustrative of why interdependency matters, and worth walking through concretely rather than abstractly. The task: fix a broken leak-scanner tool, use it to baseline a reference repo, recover lost feature branches that only exist as unreachable git objects, find and scrub leaked credentials from the entire git history (not just the working tree), and assemble a report — all while leaving ordinary project history intact and not re-cloning. The killer interdependency: removing leaked credentials from git history requires pruning the commits that contained them, but the lost teammate branches survive only as unreachable objects that git will also prune once nothing references them. An agent that scrubs credentials before recovering the lost branches destroys the lost work permanently — no later action can bring it back, because the objects are gone, not just unlinked. This is precisely a case where “keep going and try to fix it” cannot work by construction, and the paper’s checklist-progress metric correctly distinguishes a run that gets this ordering right (reaches all 7 criteria) from one that doesn’t (stalls at criterion 4, say, having already destroyed criterion-4’s prerequisite).

Quality-control pipeline (Appendix A.2). Tasks were authored by an LLM agent under a fixed protocol, then admitted only after passing three deterministic gates: the shipped forward-only reference solution must satisfy every criterion; a deliberately order-violating reference run must fail at exactly the criterion whose precondition it skips (confirming the interdependency is real, not just claimed); and the task as shipped must not already be solved (no trivially-satisfied criteria). On top of the deterministic gates, human reviewers scored each task on naturalness/fidelity/fairness dimensions (means 3.65-5.00 out of 5, all reviewed tasks accepted) and explicitly checked for contrived failure mechanisms — a reasonable, if not airtight, defense against the common benchmark-construction failure mode where an LLM-authored task turns out to reward gaming the grader rather than doing the actual work.

10. The experimental walkthrough: six results in sequence

Result 1 — Continue baselines across seven models (Table 1) establish that MettleBench isn’t saturated by any model, and that trace length and success don’t correlate simply. Task success under plain forward execution (Continue) ranges from 28.0% (Kimi K2.5) to 73.2% (Qwen3.7-Max) — no model comes close to solving the benchmark, leaving genuine headroom for a recovery mechanism to matter. The trace-length numbers are the more interesting detail: Kimi K2.5, DeepSeek-V4-Flash, and GLM-5.1 all average over 300 recorded LLM/tool events per task, while GPT-5.4 achieves the second-best success rate with the shortest average trace (50.5 events) of any model. The paper’s reading — “longer execution alone does not guarantee greater progress” — is a useful corrective against a naive intuition that more steps should mean more chances to eventually get things right; in practice, models that flail longer aren’t necessarily making more net progress, they may just be looping.

Result 2 — The four-strategy comparison (Figure 3, Table 13) is the paper’s central claim, and it’s a clean win for AgentRewind under both models tested. Under GPT-5.4: AgentRewind reaches 87.8% ± 1.2% success vs. Continue’s 62.2% ± 2.1% (a +25.6-point absolute gain, though the headline “+10.2” annotation in Figure 3 is the gain at the truncated 250-record cutoff shown in the inset, not the full-run gain — worth flagging since the two numbers get conflated if you’re skimming). Under GPT-5.4 mini (a genuinely weaker model, 33.7% baseline success): AgentRewind still reaches 51.2% ± 4.2%, a smaller absolute gain (+17.5 points) but a larger relative one (roughly 1.5×). Two of the four strategies are notably weaker than plain Continue in at least one regime: Safety Review actually underperforms Continue under GPT-5.4 (34.1% vs 62.2%) — the safety monitor is apparently blocking a meaningful number of actions the task genuinely needs, a false-positive cost that isn’t compensated by whatever true positives it catches — and Restart-with-Experiences, while beating Continue on success rate, shows “inconsistent checklist-progress gains and larger fluctuations” (visible in the wider error bars in Figure 3(c)), consistent with the intuition that discarding all progress on every restart is wasteful even when the final success number looks fine.

Result 3 — The horizon-stratified breakdown (Figure 4) is the paper’s best evidence for why AgentRewind works, not just that it works. Splitting MettleBench tasks into short/medium/long tertiles by each model’s own median Continue trace length, AgentRewind’s advantage over the best baseline is a modest +2.3-2.4 points on short-horizon tasks but balloons to +11.5 to +16.7 points on medium- and long-horizon tasks. This is exactly the pattern the paper’s core thesis predicts: short tasks don’t accumulate enough compounding error for rewind to matter much (there’s simply less to go wrong before the task ends one way or another), while long tasks are precisely where an early, uncorrected mistake has the most runway to metastasize into something unrecoverable by forward patching alone.

Result 4 — Generalization beyond MettleBench (Table 2, on the full Terminal-Bench 2.0) shows the gain isn’t an artifact of MettleBench’s specific construction. AgentRewind: 83.1% success / 90.2% average criteria passed, vs. Continue’s 78.7%/88.7% and Restart-with-Experiences’ 70.8%/79.2%. The gap here is narrower than on MettleBench — unsurprising, since Terminal-Bench 2.0’s criteria aren’t ordered/interdependent by design, so the specific “destroy irreplaceable prerequisite work” failure mode MettleBench targets is less prevalent — but AgentRewind still wins on both metrics, and Restart-with-Experiences is clearly the worst strategy here, reinforcing that “throw everything away and start over” is a costly default even outside MettleBench’s specifically-constructed interdependencies.

Result 5 — The Astropy FITS case study (Figure 5) is worth reading carefully because it’s the one place the paper shows its mechanism working (and its baselines failing) at a level of concrete detail beyond aggregate numbers. The task: repair a FITS-file library’s defects, generate a catalog covering both valid images and six intentionally malformed test fixtures (that must remain malformed so the repaired library continues correctly rejecting them), preserving those fixtures unchanged. Continue ran the catalog generator with an incorrect mode parameter; on hitting the six malformed fixtures, the generator “helpfully” auto-normalized them by overwriting the source files with valid data — permanently destroying the malformed test cases in the process. Continue’s later corrective action recreated malformed-looking files, but their contents no longer matched the originals — a subtle, silent divergence that a naive pass/fail check wouldn’t catch but MettleBench’s exact-criteria checks did (final score 9/10, one criterion permanently unsatisfiable). Restart-with-Experiences, across 26 restart attempts, recovered the original fixtures every time (since restart resets the whole workspace) but at the cost of discarding the correctly-completed library repairs on every attempt too, and never found a restart path that got all the way to 10/10 (best: 8/10). AgentRewind made the same initial mistake (ran the catalog generator with the wrong mode, reaching 8/10) — the mechanism doesn’t prevent the mistake, only lets you recover from it — but then rewound specifically to the checkpoint right before catalog generation, which retained the validated library repairs (correct, no need to redo) while restoring the original fixtures and discarding only the bad catalog-generation branch. The replacement suffix reran the generator correctly, reached 10/10, passing all 34 tests.

Result 6 — Cross-harness generality (Table 3) and the component ablation (Table 5) close the empirical loop. Table 3 shows AgentRewind’s gains hold across three structurally different agent harnesses (mini-SWE-agent: +25.6 points; FnCallAgent: +23.2; CodeAgent/smolagents: +15.8) — the smallest gain on CodeAgent, interestingly, is also the harness requiring the most bespoke plumbing (per §6 above), which is at least consistent with (though doesn’t prove) the idea that harness-specific friction in exposing the rewind mechanism costs some of its benefit. The ablation (Table 5) removes one component at a time from full AgentRewind: removing environment rewind costs the most (success drops 87.8%→43.9%, a 43.9-point collapse — larger than removing context rewind’s 21.9-point drop or rewind memory’s 36.6-point drop), confirming the paper’s framing that leaving the workspace dirty while resetting the conversation is close to useless — the agent “believes” it’s at an earlier point but the world contradicts that belief at every turn. Removing context rewind (leaving discarded actions/observations in context) is a milder but still substantial hit, plausibly because stale reasoning and false conclusions from the abandoned branch keep polluting the model’s attention even with a clean workspace. All three components clearly matter; none is redundant with the others.

11. Deployment considerations and cost

The paper doesn’t report wall-clock or token-cost overhead numbers for the recording/rewind infrastructure itself, which is a real gap for anyone trying to decide whether to adopt this — but some cost characteristics can be inferred structurally. Recording is designed to be “transparent” (Appendix C: it wraps existing client entry points rather than requiring model changes), so the recording overhead should be small — mostly the cost of writing JSON lines to disk plus computing file-diff hashes on each tool call, both cheap relative to an LLM round-trip. The real cost is in token/context budget consumed by the rewind machinery itself: every backtrack_candidates call spends a turn and returns up to 80 checkpoint summaries into context, and every accumulated rewind-memory injection grows the effective context length at every subsequent LLM call for the rest of the run. Given no rewind-per-run limit is imposed in any reported experiment (“Rewinds per run: unlimited” in Tables 10-12), a pathological task that triggers many rewinds could see meaningfully inflated per-step cost purely from memory accumulation — exactly the concern raised in §8 above regarding cumulative, never-pruned memory.

12. Limitations, in the paper’s own words and beyond them

What the paper states directly (Conclusion): “AgentRewind currently restores only controlled state and relies on external validation to identify stalled execution.” Unpacking both halves: “restores only controlled state” is the filesystem-only recovery boundary already discussed in §6 — network calls, external services, and any state outside the local workspace are permanently unrecoverable by this mechanism. “Relies on external validation to identify stalled execution” means the feedback that tells the agent something is wrong (MettleBench’s per-submission feedback naming the first unsatisfied criterion) comes from the benchmark’s evaluator, not from the agent’s own internal judgment of its progress — in a real deployment without such a ground-truth checker available at every step, the agent would need some other signal to know it should even consider rewinding, and the paper doesn’t test AgentRewind’s behavior when that external validation signal is noisy, delayed, or entirely absent.

What the paper understates or leaves unaddressed:

• Rewind decision quality is entirely delegated to the same LLM whose earlier decision presumably caused the problem, and this is never separately evaluated. The paper measures whether AgentRewind-equipped agents do better in aggregate, but never isolates whether the gain comes from good target-checkpoint selection specifically, as opposed to simply having more attempts (via the ability to retry from any point) at a task that would eventually succeed under enough forward retries anyway. A useful missing ablation: compare against “rewind to a random earlier checkpoint” as a control — if AgentRewind’s selection is only marginally better than random, most of the benefit is coming from having a recovery mechanism at all rather than from the agent’s judgment about where to recover to.
• No cost accounting (tokens, wall-clock, $) is reported anywhere, despite this being the single most practically decision-relevant number for anyone considering deployment. “Unlimited rewinds per run” is fine for a research benchmark with no cost constraint, but real deployments will need to know: how many rewinds does a typical successful run actually use, and what’s the total token cost relative to Continue for an equivalent outcome?
• The interdependency statistics reported (Table 6, Appendix A.3) confirm MettleBench’s design is interdependent, but don’t establish how often real-world engineering tasks actually have this property versus being closer to the single-primary-requirement structure the paper critiques other benchmarks for. This matters for external validity: if truly deeply-interdependent, destructively-irreversible tasks are rarer in practice than MettleBench (deliberately constructed to maximize this property) suggests, AgentRewind’s advantage in real deployments may regress toward the Terminal-Bench-2.0-sized gap (a few points) rather than the MettleBench-sized gap (25+ points).
• The paper reports a 74.7% cross-run agreement rate for identical (task, strategy, model) triples (Appendix A.5) — meaning roughly 1 in 4 repeated runs of the same configuration disagree on success/failure. This is presented as a benchmark-noise characterization, and appropriately caveated as an “upper bound… not an estimate” of evaluator noise specifically. But it also means every point estimate in Tables 1-5 carries real run-to-run variance that the reported standard deviations (computed from only $n=3$ runs) may understate, and readers should weight the smaller headline gaps (e.g., the Terminal-Bench 2.0 Table 2 numbers, which report no variance at all) with correspondingly more caution than the MettleBench headline numbers, which at least report $\pm$ std across three runs.

13. Critical analysis

Weaknesses and flaws specific to this paper. First, the paper’s strongest quantitative claims (the MettleBench Table 13 numbers, e.g., 87.8% vs 62.2%) rest on a benchmark the same research group both constructed and evaluated on, with LLM-authored task specifications reviewed by an “adversarial LLM reviewer” rather than fully independent human review at the criterion level (Appendix A.2 states human review was “at the task level,” not per-criterion) — this is a real, if increasingly common in this literature, methodological concern about the benchmark potentially being tuned (even unintentionally) toward the kind of failure mode the proposed method is designed to fix. Second, the rewind-decision-quality gap flagged in §12 is a genuine hole: without a random-checkpoint control or some other ablation isolating selection quality from availability of recovery, the paper cannot actually support the stronger claim (“the agent picks good rewind targets”) as opposed to the weaker, still-useful-but-less-impressive claim (“having any recovery option beats having none”). Third, the component ablation (Table 5) tests removing one mechanism at a time from the full system, but never tests interaction effects — for instance, whether removing rewind memory and increasing the checkpoint-candidate count from 80 to some larger number would partially compensate, which would tell us something about whether these components are truly complementary or partially redundant given enough of the other.

Limitations the authors understate or omit. The external-effects-cannot-be-undone limitation (§6, §12) is stated as a one-sentence caveat in the main text but deserves much more attention given how common external side effects are in exactly the kind of long-horizon software-engineering tasks this paper targets — pushing to a remote git repo, hitting a package registry, sending a Slack notification, calling a paid API — any of these inside a discarded branch is a permanent, silent divergence between the agent’s now-clean local workspace and the actual state of the world, and the paper offers no detection mechanism (e.g., flagging “this action might have an irreversible external effect, consider before executing” the way the Safety Review baseline flags unsafe actions) for this specific hazard, despite Safety Review already existing as a comparison point that could plausibly be adapted or combined with rewind for exactly this purpose. Second, the cumulative-memory growth concern (§8, §11) is never measured, only structurally implied — a reader has no way to know from this paper whether it’s a real problem in practice or a non-issue, and that ambiguity should have been closed with one more table.

Concrete, specific improvement suggestions. (1) Report a random-checkpoint-selection ablation and a “rewind always to the immediately-preceding checkpoint” ablation (i.e., no real target selection at all, just single-step backtracking repeated as needed) to isolate how much of the gain is from selection quality versus mere availability of recovery. (2) Report token-cost and wall-clock-cost numbers per successful/unsuccessful run for AgentRewind vs. Continue, ideally as a cost-normalized success-rate comparison (success per dollar/token), since a mechanism that wins on raw success but costs 3× the tokens is a different practical proposition than one that wins for free. (3) Add an external-side-effect flagging mechanism — even a simple heuristic classifier distinguishing “local, reversible” from “external, irreversible” tool calls, surfaced to the agent before execution — and measure whether combining it with rewind changes behavior around genuinely irreversible actions. (4) Report rewind-memory block length over the course of a run (mean/max tokens consumed by $M$ at rewind $j$) to directly address whether the cumulative-and-never-pruned design (§8) is a latent context-budget problem waiting to manifest on harder or longer-running tasks than those in MettleBench’s 82-task set.

14. Reproducibility notes

The paper provides two public repositories: code at github.com/Futuresis/replay-agent-recorder and the MettleBench dataset at github.com/Kelvin-Coffee/MettleBench. Appendix B gives fully specified experimental configurations (temperature, step/wall-clock limits, termination conditions, rewind-specific parameters like candidate count and memory accumulation mode) in tabular form for every reported comparison, which is good practice and should make the headline Table 1/13 numbers directly re-runnable given API access to the seven evaluated base models. The evaluator determinism audit (Appendix A.5) — confirming none of the 82 task evaluators draws random numbers, reaches the network, or generates non-deterministic identifiers in its judgment — is a genuinely useful piece of due diligence that strengthens confidence the reported success/failure labels are at least evaluator-deterministic (modulo the separately-reported 74.7% cross-run agent-behavior agreement rate, which is a different source of variance entirely).

15. Closing take

AgentRewind is a well-executed instance of importing a decades-old systems idea — checkpoint and restore — into a domain (LLM agent execution) where the naive version of the idea (just checkpoint the conversation, or just checkpoint the filesystem) doesn’t actually solve the problem, because the two halves of “agent state” have to be kept jointly consistent to be useful, and because what makes rewind valuable rather than merely safe is the memory that travels forward with it. The MettleBench contribution is arguably as important as the mechanism itself: it’s a benchmark specifically engineered to have the property (destructive, order-dependent operations on shared state) that makes recovery mechanisms actually matter, filling a real gap left by single-requirement benchmarks. The empirical story is consistent and the mechanism’s operating principle (retain validated prefix, discard only the degraded suffix, carry forward what you learned) is intuitive enough to explain to a non-specialist in one sentence — always a good sign for whether an idea will actually get adopted rather than just cited. The open questions that matter most for real deployment — cost, selection-quality vs. mere-availability, and what to do about genuinely external, irreversible side effects — are exactly the questions a follow-up paper (or an engineering team building on this code) should tackle next.