Long-Horizon-Terminal-Bench: Dense Reward Grading Exposes Agent Long-Horizon Execution Limits

Review date: 2026-07-13 Review author: Zhongzhu Zhou Paper reviewed: Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading Paper authors: Zongxia Li†, Zhongzhi Li†, Yucheng Shi†, Ruhan Wang, Junyao Yang, Zhichao Liu, Xiyang Wu, Anhao Li, Yue Yu, Ninghao Liu, Lichao Sun, Haotao Mi, Leowei Liang arXiv: https://arxiv.org/abs/2607.08964 Venue/Status: Preprint (arXiv, July 2026) — Tencent HY LLM Frontier + UMD, UGA, UMN, IU, Lehigh, NUS, PolyU

Short Answer

Long-Horizon-Terminal-Bench (LHTB) is a 46-task benchmark for AI agents operating in containerized terminal environments, designed to stress two capabilities that short-horizon benchmarks barely touch: long-horizon planning (tasks requiring hundreds of episodes and tens of minutes to hours of execution) and progress verification (knowing whether you are done). Rather than scoring agents binary pass/fail, every task is decomposed into semantically meaningful subtasks each with its own grader, and the final score is a weighted average — a dense reward that quantifies how far an agent actually progressed.

The paper evaluates 15 frontier models under a shared Terminus-2 harness. Even the best model (GPT-5.5) resolves only 15.2% of tasks at a 0.95 reward threshold, and the mean pass rate across all models is 4.3%. Failure analysis reveals that 79% of all failed runs hit the 90-minute timeout while still actively working — not because every local step is wrong, but because agents cannot reliably turn sustained local progress into finished long-horizon artifacts within budget. An additional 19% of failures are “early exits” where agents stop voluntarily before satisfying the hidden verifier, a pattern the paper calls false finishes that implies systematic overconfidence in self-verification. Dense rewards are essential to reveal both patterns: binary grading collapses all partial-progress runs into the same failure bucket, hiding the mechanistic difference between a model that gives up early and one that works hard but runs out of time.

Prerequisites: What You Need to Understand This Paper

1. LLM Agents and Terminal Environments

An LLM agent is a language model that takes actions in an environment by issuing tool calls or shell commands, receiving observations (output, file listings, errors), updating its context, and repeating. Terminal-based agents specifically operate inside a containerized shell: they can read and write files, run scripts, inspect partial outputs, and call system utilities — exactly what a human software engineer does when working through a long project.

The key scaling challenge is that each interaction is a new context window chunk. As the task grows longer, the agent must maintain coherent state, update its plan, avoid revisiting solved steps, and know when it is done. None of these are trivial for current models.

2. Benchmark Taxonomy: Short- vs. Long-Horizon

Short-horizon benchmarks (SWE-Bench, HumanEval, LiveCodeBench) evaluate whether an agent can solve a well-scoped, self-contained task in a few steps or minutes. They are valuable, but they mostly measure whether an agent can reason correctly given a crisp, stationary problem.

Long-horizon benchmarks add a time dimension: tasks require hundreds of steps, multi-stage planning, error recovery across many sub-goals, and iterative debugging. Completing them requires not just reasoning but budget management — the ability to allocate effort appropriately across sub-goals so that the total progress is maximized within a time budget.

3. Sparse vs. Dense Reward Signals

A binary reward gives 1 if the agent completes the final goal, 0 otherwise. For hard tasks where almost no model passes, this is nearly pure noise: every failure looks the same regardless of how far the agent got. Dense rewards assign partial credit to intermediate steps, making it possible to distinguish an agent that solved 80% of the task from one that solved 5%, even if neither passes.

In reinforcement learning, the difference between sparse and dense rewards is the difference between an unlearnable and a learnable environment for the same objective. In evaluation, the analogy is direct: sparse rewards cannot distinguish model capability at a fine granularity when all models fail to reach the final goal.

4. Docker / Containerization

Each LHTB task runs inside a Docker container that is pre-loaded with all relevant files, code, data, tools, and helper scripts. The container provides complete isolation: the agent’s actions cannot affect the host environment. Graders run inside the container at the end of each rollout, inspecting the container’s final state. This deterministic setup makes evaluation reproducible.

5. Agent Harnesses

An agent harness is the scaffolding around a base LLM that provides: (a) environment interaction loop (send command → get output → append to context → generate next command); (b) context management (truncation, summarization); (c) stopping criteria (budget, self-termination). Two harnesses are used: Terminus-2 for most models, and OpenAI’s Codex harness for GPT-5.3 Codex. Harness differences can affect performance substantially, independent of the underlying model.

6. Pass@1 and Mean Reward

Two primary metrics appear throughout:

• Pass@1(τ): fraction of 46 tasks where the final reward R ≥ τ, for threshold τ ∈ {0.9, 0.95, 1.0}.
• Mean reward R̄: average of final reward values across all 46 tasks, capturing partial progress even on tasks the model does not pass.

These measure different things. Pass@1 answers “does the agent finish tasks reliably?” Mean reward answers “how far does the agent typically get?” High mean reward with low pass@1 signals near-complete performance that falls just short of the threshold — a useful leading indicator of progress.

Motivation: What is Missing in Existing Benchmarks

The Gap Between Benchmark Performance and Real-World Workflows

By mid-2026, frontier models score impressively on SWE-Bench and Terminal-Bench. But real professional workflows look very different:

• Reproducing the results of a published ML paper requires installing dependencies, fixing version mismatches, finding the right hyperparameters, running training for hours, checking that checkpoints match expected metrics, and generating the figures — not 10 steps, but hundreds.
• Auditing a multimodal dataset requires inspecting thousands of samples, running quality-control scripts, diagnosing systematic errors (rotated frames, schema mismatches, corrupted audio), and producing a clean output — open-ended and requiring long-horizon iteration.
• Debugging a robotics SLAM pipeline with renamed fields, missing values, and injected noise requires systematic diagnosis followed by end-to-end repair and verification.

None of these fit the “crisp problem, 10-step solution” paradigm of existing benchmarks. They require what LHTB calls long-horizon execution: hundreds of episodes, sustained progress across multiple sub-goals, and reliable verification at the end.

The Sparse Reward Problem at Extreme Difficulty

At the difficulty level of LHTB, outcome-only grading is nearly useless for model discrimination. Under R ≥ 1.0 (perfect score), 10 of the 15 evaluated models pass exactly zero tasks. Binary grading cannot tell these models apart at all. Dense rewards resolve this: the mean reward (R̄) for these same 10 models spans from 0.08 (Grok 4.20) to 0.32 (GLM 5.2), a 4× spread that captures real capability differences.

The paper makes a clean argument: sparse rewards are only informative when tasks are not too hard. Once benchmark difficulty exceeds model capability such that the pass rate collapses toward zero, binary grading loses all discriminative power. Dense subtask rewards rescue discriminability.

Benchmark Design

2.1 Task Formulation

LHTB inherits the Terminal-Bench task format (Harbor tasks) and extends it. Each task is specified by:

1. Natural-language instruction: describes the overall goal; the only specification the agent sees.
2. Docker image: pre-loaded with all assets, code, data, and tools.
3. Task configuration file: metadata, time limits, harness parameters.
4. Oracle implementation or simulator: used by the grader to produce ground-truth answers or run reference solutions.

The key departure from Terminal-Bench is the grading scheme: instead of a single pass/fail check at the end, each task is additionally decomposed into K semantically meaningful subtasks {s₁, …, sK}, each with its own deterministic checker.

2.2 The Core Reward Formula

The final task reward is a normalized weighted sum of subtask scores:

$$R = \frac{\sum_{k=1}^{K} w_k \, r_k}{\sum_{k=1}^{K} w_k} \tag{1}$$

where:

• $K$ is the number of subtasks in the task,
• $w_k \geq 0$ is the weight of subtask $k$ (reflecting its relative importance),
• $r_k \in [0, 1]$ is the score for subtask $k$ from the deterministic grader.

By default, all weights are equal ($w_k = 1/K$), which simplifies to a plain average:

$$R = \frac{1}{K} \sum_{k=1}^{K} r_k \tag{2}$$

When the final goal is substantially more important than intermediate checkpoints, the paper increases $w_K$ to concentrate reward on end-to-end completion while still giving partial credit for reaching intermediate milestones.

Why this formula? A plain average is unbiased and easy to interpret. Agents can neither “hack” a single checkpoint to inflate their score nor be unfairly penalized for completing 95% of a task. The weighted variant accommodates tasks where sub-goals are not equally important (e.g., a final pipeline must produce a correct output to be useful, even if intermediate diagnostics were done correctly).

2.3 Subtask Types

Three subtask types are defined, each implementing $r_k$ differently:

Type 1 — Binary subtasks. The grader runs a programmatic condition on the container state. Examples: all unit tests pass, a service responds on the expected port, required experiment scripts complete without error.

$$r_k = \mathbf{1}[\text{condition}(s)] \in \{0, 1\} \tag{3}$$

Type 2 — Continuous or thresholded subtasks. For quantitative targets (reproducing a metric within tolerance $\varepsilon$, achieving a given speedup), the score decays linearly from 1 to 0 as the deviation from the reference grows:

$$r_k = \max\!\left(0,\; 1 - \frac{|\hat{y}_k - y_k|}{\varepsilon_k}\right) \tag{4}$$

where $\hat{y}_k$ is the agent’s output value and $y_k$ is the reference. This gives full credit for exact matches, partial credit for near-misses, and zero for outputs far from the reference.

Type 3 — Episode-aggregating subtasks. Campaign-style tasks (games, repeated audits) require agents to perform a long-horizon behavior reliably across $N$ episodes. The subtask score is the fraction of episodes in which the environment’s internal success flag fires:

$$r_k = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}[\text{success}_i] \tag{5}$$

This measures reliability rather than lucky single-trial success.

Pseudocode for the LHTB Grading Algorithm:

Algorithm 1: LHTB Task Grader
Input:  final container state S, task definition (w_1..K, grader_1..K), threshold τ
Output: task reward R ∈ [0, 1], pass flag

1:  for k = 1 to K do
2:      if task.type[k] == BINARY:
3:          r_k ← 1 if grader_k(S) else 0
4:      else if task.type[k] == CONTINUOUS:
5:          ŷ_k ← extract_value(S, grader_k.target)
6:          r_k ← max(0, 1 - |ŷ_k - y_k| / ε_k)
7:      else if task.type[k] == EPISODE_AGGREGATE:
8:          successes ← 0
9:          for i = 1 to N do
10:             successes += 1 if grader_k.episode_check(S, i) else 0
11:         r_k ← successes / N
12:     end if
13: end for
14: R ← Σ_k (w_k * r_k) / Σ_k w_k   # Eq. (1)
15: pass ← (R >= τ)
16: return R, pass

The grader runs once per rollout at the very end, inside the container. The container state S captures the entire file system, running processes, and any simulator outputs. Subtask scores are additive but never more than 1.0 total.

2.4 Pass@1 and Mean Reward

The primary evaluation metrics are:

$$\text{pass@1}(\tau) = \frac{1}{|T|} \sum_{t \in T} \mathbf{1}[R_t \geq \tau] \tag{6}$$ $$\bar{R} = \frac{1}{|T|} \sum_{t \in T} R_t \tag{7}$$

where $T$ is the set of 46 tasks. Three thresholds are reported: $\tau \in \{0.9, 0.95, 1.0\}$. The paper treats $\tau = 0.95$ as the primary threshold (relaxed enough to give credit for near-perfect runs while excluding clearly incomplete ones).

Figure 1: LHTB Evaluation Pipeline

flowchart TD
    A["Natural Language Instruction\n(only visible to agent)"] --> B

    subgraph DockerContainer["Docker Container (isolated)"]
        B["Agent receives observation\n(shell output, files, errors)"]
        B --> C["LLM generates next action\n(shell command / file edit)"]
        C --> D["Action executes in terminal"]
        D --> E{"Budget expired\nor agent exits?"}
        E -- No --> B
        E -- Yes --> F["LHTB Grader runs\ninside container"]
    end

    F --> G["Subtask scores r_1...r_K"]
    G --> H["Weighted average → R = Σw_k r_k / Σw_k"]
    H --> I{"R ≥ τ?"}
    I -- Yes --> J["Pass@1 +1"]
    I -- No --> K["Partial progress recorded\nin mean reward"]

    style A fill:#e8f4f8
    style F fill:#ffd700
    style H fill:#d4edda

2.5 Dataset Construction Process

Pseudocode for LHTB Task Construction:

Algorithm 2: LHTB Task Construction
Input:  a real professional workflow domain W
Output: a Harbor-format task with hidden verifier

Phase 1 — Seed workflow
1:  Identify a realistic, long-horizon professional problem in W
    (e.g., NetCDF schema mismatch, SLAM benchmark repair, figure data recovery)

Phase 2 — Build a deliberately broken project
2:  Implement a complete but broken terminal project:
    a. Write a weak baseline implementation (partial, easy to pass visible tests)
    b. Write an official gold solution (achieves verifier score 1.0)
    c. Create a multi-step solve.sh (gold solution workflow)
    d. Inject systematic breaks:
       - Renamed fields, missing values, injected noise
       - Gzip/base64 encoding, schema aliases
       - Rotated/cropped images, anomalous frames

Phase 3 — Design hidden verifier
3:  Write a grader with:
    a. Public checks: CLI behavior, file formats, simple examples (low weight, visible)
    b. Hidden stress suite: harder inputs, schema variations, edge cases (high weight, invisible)
    c. Calibrate so that gold solution scores exactly 1.0

Phase 4 — Calibration loop
4:  Repeatedly run DeepSeek-V4-Pro under 1.5-hour budget
5:  Adjust task difficulty until tasks are:
    - Not trivially solvable (model does not always pass)
    - Not completely impossible (model can make partial progress)

Phase 5 — Package as Harbor task
6:  Create task.yaml, Dockerfile, README.md, data/, scripts/, tests/, solve.sh
7:  Add to benchmark with containerized environment + Terminus-2 agent harness

The hidden verifier design is crucial: it prevents agents from gaming visible public tests by hard-coding outputs or patching only the simple cases. Robust solutions must generalize to hidden schema variants and stress cases that are never revealed during the agent’s rollout.

Figure 2: Task Category Distribution

pie title LHTB Task Distribution (46 total)
    "Software & Reverse Engineering (7)" : 7
    "Earth, Climate & Energy (6)" : 6
    "Multimodal & Imaging Analysis (6)" : 6
    "Scientific Computing & Simulation (6)" : 6
    "Research Reproduction & ML (5)" : 5
    "Systems, Performance & Security (5)" : 5
    "APEX Professional Workflows (4)" : 4
    "Interactive Games (4)" : 4
    "Logic & Constraint Puzzles (3)" : 3

The 46 tasks are spread evenly across nine domains — no single domain dominates, which prevents strong domain specialists from achieving high pass rates without general long-horizon capability.

Experiments

3.1 Models and Harnesses

Models evaluated (15 total): GPT-5.5, GPT-5.4, GPT-5.3 Codex (via OpenAI Codex harness), DeepSeek V4 Pro, Gemini 3.1 Pro, GLM 5.1, GLM 5.2, Kimi K2.6, Kimi K2.7 Code, MiniMax M3, Qwen3.7 Max, Qwen3.6 Plus, Doubao Seed 2.1 Pro, Hy3, Grok 4.20.

Harnesses:

• Terminus-2: standard harness for 14 of 15 models; provides a long-horizon terminal session with a consistent prompt template.
• OpenAI Codex: used for GPT-5.3 Codex; provides more agentic scaffolding designed for coding tasks.

Harness differences mean that comparisons between GPT-5.3 Codex and the Terminus-2 group are not perfectly controlled — a limitation the paper acknowledges.

Budget: 90 minutes per task per model. Tasks that exceed this budget are terminated as timeouts.

Metrics reported per model:

• Pass@1 at τ ∈ {0.9, 0.95, 1.0}
• Mean reward R̄
• Average tokens/task, episodes/task, execution time/task
• Estimated cost per task (USD)

3.2 Average Resource Usage

Across all models, the average per-task statistics are striking:

Metric	Average
Episodes per task	231
Tokens per task	9.9M
Execution time	85.3 min
Cost per task	~$10.2

These numbers are substantially larger than Terminal-Bench 2 (most trials finish under 20 minutes). The 85-minute average out of a 90-minute budget suggests most models nearly exhaust the budget on every task, which is direct evidence that long-horizon completion — not local reasoning — is the binding constraint.

Main Results

4.1 The Leaderboard

The performance ordering at τ = 0.95 (primary metric):

Model	Pass@1 (R≥0.95)	Mean R
GPT-5.5	15.2% (7/46)	0.44
MiniMax M3	6.5% (3/46)	0.39
Kimi K2.7 Code	6.5% (3/46)	0.37
DeepSeek V4 Pro	6.5% (3/46)	0.32
Qwen3.7 Max	4.3% (2/46)	0.31
Doubao Seed 2.1 Pro	4.3% (2/46)	0.30
Gemini 3.1 Pro	4.3% (2/46)	0.29
GLM 5.1	4.3% (2/46)	0.28
GPT-5.3 Codex	4.3% (2/46)	0.29
GLM 5.2	2.2% (1/46)	0.32
Qwen3.6 Plus	2.2% (1/46)	0.31
GPT-5.4	2.2% (1/46)	0.27
Hy3	2.2% (1/46)	0.25
Kimi K2.6	0% (0/46)	0.25
Grok 4.20	0% (0/46)	0.10

Several observations:

GPT-5.5 leads, but by a wide margin over the pack. At 15.2%, it is more than twice the next tier (6.5%). This suggests GPT-5.5 has a qualitatively different capability for long-horizon task completion — not just marginally better local reasoning, but better horizon management.

At τ = 1.0 (perfect score), only 5 models pass even one task, and 10 models pass zero. This near-total collapse at τ = 1.0 reveals how fragile full completion is: even getting 95% of the way through a task but failing the final hidden stress test (strict numerical tolerance, edge-case schema, etc.) results in zero credit under binary grading.

Dense rewards rescue discriminability. Mean reward spans 0.10 (Grok 4.20) to 0.44 (GPT-5.5), a 4.4× range even when pass rates cluster near zero. GLM 5.2 achieves a mean reward of 0.32 despite passing only one task, outperforming models with higher pass rates like GPT-5.4 (mean R = 0.27).

Figure 3: Reward Distribution Across All Runs

xychart-beta
    title "Distribution of final rewards (15 models × 46 tasks = 690 runs)"
    x-axis ["<0.05", "0.05-0.15", "0.15-0.25", "0.25-0.35", "0.35-0.45", "0.45-0.55", "0.55-0.65", "0.65-0.75", "0.75-0.85", "0.85-0.95", "≥0.95"]
    y-axis "Number of runs" 0 --> 230
    bar [224, 117, 66, 55, 37, 24, 38, 54, 20, 23, 30]

Key statistics from this distribution:

• 30 runs (4.4%) pass at R ≥ 0.95
• 224 runs (32.6%) make no meaningful progress (R < 0.05)
• 433 runs (62.8%) achieve partial reward in [0.05, 0.95) — real progress that binary grading would hide
• 180 runs (26.1%) reach R ≥ 0.5
• Near-complete runs (0.85 ≤ R < 0.95): 23 runs — nearly as many as full passes

The bimodal shape — a large pile at R < 0.05 and a smaller cluster around R ≥ 0.5 — tells us something important: when frontier models attempt LHTB tasks, they either make very limited progress (< 5%) or make substantial progress (> 50%). There is less mass in the middle range than one might expect from a smooth difficulty spectrum. This suggests that early task understanding and setup is critical: models that get the task right in the first few steps often make it most of the way; models that misunderstand the task in early steps stall quickly.

4.2 Dense Rewards vs. Binary Grading: Why It Matters

The paper makes this argument formally with a simple example: under binary pass/fail at R ≥ 1.0, 10 of 15 models are indistinguishable (all score 0). With mean reward, the same 10 models span from 0.10 to 0.32.

More concretely, consider Kimi K2.6 (0 tasks passed at R ≥ 0.95, mean R = 0.25) vs. Grok 4.20 (0 tasks passed, mean R = 0.10). Under binary grading, they are tied. Under dense rewards, Kimi K2.6 is 2.5× more capable in terms of partial progress. Kimi K2.6 has five near-misses (R ≥ 0.75), including grammar-fuzz-coverage-hunt (R = 0.94), spot-scheduler-traces (R = 0.90), poc-exploit-craft (R = 0.89), and nbody-accel-iterative (R = 0.89). These are near-complete solutions that fail only the final hidden checks.

Figure 4: Dense Rewards vs. Binary Grading — Comparison

flowchart LR
    subgraph BinaryGrading["Binary Grading (R ≥ 1.0)"]
        direction TB
        B1["GPT-5.5: 10.9% pass"]
        B2["MiniMax M3: 0% pass"]
        B3["Kimi K2.6: 0% pass"]
        B4["Grok 4.20: 0% pass"]
        B5["GLM 5.2: 0% pass"]
        Note1["⚠ 10 models all\nscore 0 — indistinguishable"]
    end

    subgraph DenseGrading["Dense Rewards (mean R)"]
        direction TB
        D1["GPT-5.5: mean R = 0.44"]
        D2["MiniMax M3: mean R = 0.39"]
        D3["Kimi K2.6: mean R = 0.25"]
        D4["Grok 4.20: mean R = 0.10"]
        D5["GLM 5.2: mean R = 0.32"]
        Note2["✓ 4.4× spread distinguishes\nmodel capability clearly"]
    end

    BinaryGrading -->|"LHTB's dense reward\nreplaces this"| DenseGrading

Failure Mode Analysis

5.1 Three Failure Types

The paper categorizes each failed run (R < 0.95) into:

1. Timeout: the agent is still working when the 90-minute budget expires.
2. Early exit: the agent terminates voluntarily before the budget expires, despite not satisfying the hidden verifier.
3. Harness error: a non-timeout exception in the agent–environment loop (API failures, verifier-side errors).

Distribution across all models: 79% timeout, 19% early exit, 3% harness error.

This distribution is the paper’s most important empirical finding. If the dominant failure were “wrong local steps” (bad reasoning), we would expect to see many harness errors or very low mean rewards across the board. Instead:

• Timeouts dominate at 79%, with mean rewards ranging from 0.10–0.35 across models — agents are making meaningful progress but running out of time.
• Early exits account for 19%, often at non-trivial rewards (Kimi K2.7 Code: mean exit R = 0.51, MiniMax M3: mean exit R = 0.42) — agents stop convinced they are done, even though the hidden verifier disagrees.

5.2 The False Finish Pattern

A false finish is defined as an early exit at R ≥ 0.75 — an agent that has completed most of the task but misjudges completion and stops with time remaining.

The paper identifies 14 such runs. Examples:

• Kimi K2.7 Code stops on duckdb-optimizer-closure at R = 0.92 with ~20 minutes remaining.
• GLM 5.2 stops on apex-ib244-matter at R = 0.90.
• Seven different models stop on apex-law433-matter between R = 0.80 and 0.87, each with ~20 minutes remaining.

In all of these, the agent has satisfied the visible public tests but not the hidden stress suite. The agent does not know the hidden tests exist (by design), so it cannot verify against them. It runs the visible tests, sees them pass, and concludes “I am done.” This is a systematic gap in self-verification capability.

Figure 5: Failure Mode Composition by Model

xychart-beta
    title "Unresolved runs by failure type per model (of 46 tasks)"
    x-axis ["GPT-5.5", "MiniMax M3", "Kimi K2.7", "DeepSeek V4", "Qwen3.7", "GLM 5.1", "Gemini3.1", "GPT-5.4", "GLM 5.2", "Qwen3.6", "GPT-5.3", "Kimi K2.6", "Hy3", "Doubao", "Grok 4.20"]
    y-axis "Unresolved runs" 0 --> 50
    bar [28, 38, 30, 26, 33, 36, 36, 32, 33, 33, 29, 42, 39, 40, 46]

The key insight from Figure 5: Grok 4.20 has 46/46 unresolved runs — it fails every single task. But its mean reward is not zero (R̄ = 0.10); it makes some partial progress on most tasks. This distinguishes Grok 4.20’s failure mode (poor but not zero progress) from a model that crashes immediately on every task.

5.3 Long-Horizon Completion vs. Local Reasoning

The paper draws a crisp distinction between two axes of agent capability:

1. Local reasoning: Can the agent execute a single step correctly? Do actions correspond to a sensible plan?
2. Long-horizon completion: Can the agent sustain progress across hundreds of steps, manage context, avoid backtracking, and finish within a time budget?

Terminal-Bench 2 primarily measures (1). LHTB measures both, and the data suggests (2) is the binding constraint at current frontier model capability levels. Agents on LHTB frequently fail not because their local reasoning is broken, but because they:

• Burn time on unnecessary verification loops,
• Repeat steps they have already completed,
• Fail to track which sub-goals remain,
• Run out of time with most of the task complete.

This is a different kind of failure than “wrong answer,” and it implies different improvements are needed: better memory/progress tracking, more efficient exploration, and more calibrated stopping criteria.

Cost Analysis

6.1 Per-Task Economics

Table 1 in the paper reports estimated cost (from public API prices as of June 2026):

Model	Tokens/task (M)	Time (min)	Cost/task ($)
GPT-5.5	4.16	72.9	21.46
GPT-5.4	10.90	79.3	27.57
GPT-5.3 Codex	4.57	80.7	8.20
DeepSeek V4 Pro	14.45	83.6	6.32
MiniMax M3	20.20	90.0	6.13
Kimi K2.7 Code	8.54	85.4	8.31
Gemini 3.1 Pro	3.55	85.0	7.61
GLM 5.1	5.84	92.6	5.13
Qwen3.7 Max	6.13	83.5	7.78
Hy3	17.21	91.3	2.47
Kimi K2.6	10.27	92.5	9.94
Grok 4.20	16.23	69.5	20.63

Key observations:

1. GPT-5.4 is the most expensive (~$28/task) but underperforms GPT-5.5 (~$21/task). This happens because GPT-5.4 requires more episodes (302 vs. 208) at comparable token prices — more episodes = more context management overhead = more tokens per task despite equivalent per-token pricing.
2. Hy3 is the cheapest at ~$2.5/task (low token price, modest episodes), and sits on the Pareto frontier despite low pass rates.
3. MiniMax M3 achieves 6.5% pass@1 (matching DeepSeek V4 Pro) at ~$6/task vs. DeepSeek V4 Pro's$6.3/task — effectively tied on the cost frontier.

6.2 The Pareto Frontier

The cost-reward Pareto frontier (minimizing cost while maximizing pass rate) passes through:

• Hy3: lowest cost (~$2.5), minimal performance (2.2% pass@1, R̄ = 0.25).
• MiniMax M3 and Doubao Seed 2.1 Pro: intermediate cost ($5–6), 4.3–6.5% pass@1.
• GPT-5.5: highest cost ($21), highest performance (15.2%).

A striking finding is that higher spending does not guarantee better performance: GPT-5.4 (~$28) achieves only 2.2$6) achieves 6.5%. The difference is efficiency: a less capable model that needs 3× the episodes to make the same progress will cost more, not less.

What LHTB Tells Us About Frontier Agent Capability

The Long-Horizon Gap Is Real and Large

At 4.3% mean pass rate across 15 frontier models (τ = 0.95), LHTB reveals a substantial unsolved capability gap for long-horizon terminal agents. Short-horizon benchmarks have become saturated (SWE-Bench Verified scores above 70% for top models), but LHTB shows that extending the horizon by an order of magnitude collapses success rates to near-zero for the same models.

The Binding Constraint is Not Local Reasoning

Since 79% of failures are timeouts — not early failures or harness errors — the limiting factor is horizon management: allocating attention and effort correctly across hundreds of steps within a fixed budget. Local reasoning improvements (better code generation, more accurate file reading) are unlikely to unlock large gains on LHTB without concurrent improvements in:

• Progress tracking: knowing which sub-goals are done vs. remaining,
• Redundancy avoidance: not re-doing steps already completed,
• Early stopping calibration: knowing when to stop exploring and commit to a solution,
• Context compression: managing a growing context window without losing critical state.

Self-Verification Is a Fundamental Gap

The false-finish pattern (14 runs with R ≥ 0.75 where the agent exits early) indicates that current agents systematically overestimate task completion when only visible tests are available. This is particularly concerning because real-world deployments rarely have perfect verifiers — the agent must judge completion from first principles. Improving agents’ ability to construct adversarial self-tests (beyond the visible public checks) and verify against them before exiting is an important open problem.

Critical Assessment: Weaknesses and Improvements

Weaknesses and Flaws

W1 — Small task count and category imbalance. LHTB has 46 tasks. This is sufficient to produce statistically meaningful rankings, but individual task variances are high: flipping 2–3 tasks changes a model’s pass@1 by 4–6 percentage points. For a benchmark at this difficulty level, variance in the score estimate itself is a concern. The interactive games category has only 4 tasks, logic puzzles only 3. The paper does not report per-category pass rates, making it impossible to determine whether a model’s overall score reflects broad competence or luck on specific categories.

W2 — Single harness for all models (mostly). 14 of 15 models use Terminus-2; GPT-5.3 Codex uses OpenAI’s own Codex harness. The paper acknowledges this but the comparison between GPT-5.3 Codex and Terminus-2 models is confounded. More broadly, no sensitivity analysis shows how much performance varies with harness choice for the same model. Given that harness quality (context management, prompting, action selection) is known to matter substantially for agent benchmarks, the single-harness design conflates model capability with harness quality. A model with a weak harness may score far below its true capability.

W3 — No ablation on grading design. The paper chooses subtask weights and grading thresholds (e.g., τ = 0.95 as the primary threshold) without ablations showing how sensitive rankings are to these choices. What happens if τ = 0.85 or τ = 0.9? Does the model ordering change? The sensitivity of pass@1 rankings to the grading design is not studied.

W4 — Difficulty calibration is model-dependent. Tasks were calibrated using DeepSeek-V4-Pro specifically (repeatedly run at 1.5-hour budget and adjusted until “challenging but solvable”). This means LHTB is implicitly calibrated to DeepSeek-V4-Pro’s capability level. A future, more capable model might find LHTB too easy; a less capable model (smaller parameter count) would find it completely unsolvable. The benchmark does not include an easy tier or a difficulty gradient that would allow it to remain informative across a wider range of model capability.

W5 — Hidden verifier transparency. The paper argues that hidden verifiers prevent gaming, which is correct. But readers cannot fully trust a benchmark whose graders they cannot inspect. The paper mentions releasing the evaluation harness, but the completeness of the public release (whether all hidden stress tests are included or only the framework) is unclear.

W6 — No multi-run variance. All results are pass@1 (a single run per model per task). At these low pass rates, single-run variance is substantial. For a model achieving 4.3% pass@1 (2/46), the 95% confidence interval under a binomial model is roughly [0.5%, 14.6%] — which overlaps with models at 2.2% and 6.5%. The paper does not provide confidence intervals, making it difficult to determine which differences in the leaderboard are statistically significant.

Limitations the Authors Understate or Omit

L1 — Terminal-only task coverage is narrowly scoped. LHTB focuses exclusively on terminal-based workflows. Many important long-horizon agent tasks involve web browsing, GUI interaction, API calls, or multi-modal reasoning in non-terminal settings. The benchmark’s findings about “long-horizon completion being the binding constraint” may not generalize to other deployment modalities.

L2 — Time budget vs. actual real-world deployment. The 90-minute budget is reasonable for a benchmark but does not reflect all real deployment constraints. Some professional workflows take days; others have hard real-time requirements. The paper’s claim that LHTB “reflects real professional workflows” is partially true but ignores that real deployments have very different time/cost profiles from a fixed 90-minute containerized evaluation.

L3 — Static benchmark vs. contamination risk. The benchmark is static (fixed 46 tasks). As frontier models are trained on more data and as the benchmark becomes well-known, contamination risk (models memorizing solution patterns) grows. The paper does not discuss contamination mitigation strategies or plans for benchmark evolution.

L4 — Cost estimates may be unreliable. Cost estimates are based on public list prices as of June 2026 and assume no caching discounts. In practice, providers offer volume discounts and prompt caching, which can significantly reduce real costs (especially for long context). The cost analysis should be treated as an approximation, not a precise efficiency ranking.

Concrete Improvement Suggestions

I1 — Scale to 200+ tasks with stratified difficulty tiers. A larger, stratified benchmark (easy/medium/hard tiers) would dramatically reduce variance and make the benchmark informative across a wider range of model capability. The easy tier could be drawn from current short-horizon benchmarks; the medium tier from the current LHTB range; the hard tier from tasks requiring multi-day workflows.

I2 — Provide multi-harness evaluation. Running each model with at least two harnesses (or computing model × harness interaction effects) would separate model capability from harness quality, giving more interpretable results. The paper could use a “canonical” harness for all models and report raw harness-controlled scores.

I3 — Add per-category and per-difficulty-tier breakdowns. Knowing that GPT-5.5 achieves 15.2% overall but fails all games tasks vs. succeeding on 40% of software engineering tasks would reveal category-specific weaknesses and guide model improvement.

I4 — Report multi-run pass@k and confidence intervals. Running each model 3 times on each task and reporting pass@3 and confidence intervals would give a much more reliable ranking at low pass rates. The cost of 3 runs at $10/task/model is$1380 per model — high but tractable for a frontier model evaluation.

I5 — Study the agent self-verification problem directly. The false-finish analysis is tantalizing but shallow. A deeper study would: (a) present agents with their own passing-visible-tests state and ask them to generate adversarial test cases; (b) compare agents that have this reflection step enabled vs. disabled; (c) measure how often the agent-generated adversarial tests catch the final hidden failures. This would directly characterize and improve the self-verification gap.

I6 — Analyze intra-task progress trajectories. The current paper only reports final reward. A richer analysis would plot reward as a function of episode count, identifying whether agents plateau early (fail to continue making progress) or make steady linear progress but run out of time. This distinction has very different implications for which improvements are most impactful.

Limitations and Boundary Conditions

Scope: LHTB measures long-horizon terminal agent capability at the 90-minute/~231-episode scale. Findings do not directly extend to:

• Very short tasks (< 10 steps): different failure modes dominate.
• Tasks requiring web browsing or GUI: terminal benchmarks cannot measure these.
• Tasks where the final objective has no deterministic verifier (open-ended writing, research ideation).

Generalization: The finding that “long-horizon completion is the binding constraint” is specific to this task distribution and time budget. At shorter time budgets, local reasoning errors may dominate. At much longer budgets (hours to days), entirely different bottlenecks (context compression, long-term memory) would emerge.

Reproducibility Notes

• Benchmark release: the LHTB tasks, Terminus-2 harness, and evaluation code are to be released (project page: https://zli12321.github.io/LHTB/).
• Docker containers: all tasks are containerized, ensuring fully reproducible evaluation environments.
• Cost estimates: based on public list prices as of June 2026; actual costs will vary with provider and time.
• Model versions: exact model versions used are not specified for all models in the paper, which could affect reproducibility as providers update models over time.
• Calibration dataset: the 120 candidate tasks (before quality filtering to 46) are not publicly released.

Conclusion

Long-Horizon-Terminal-Bench is a timely and carefully designed benchmark that exposes a clear capability gap in current AI agents: they can reason well locally but cannot reliably complete long-horizon workflows within a practical time budget. The dense subtask-based grading scheme is its most important methodological contribution — it transforms a nearly-uninformative binary measurement at this difficulty level into a rich, model-discriminating signal. The finding that 79% of failures are timeouts (not reasoning errors) and that 19% are false finishes (not lack of capability) is a precise, actionable diagnosis: the next wave of agent improvements needs to focus on horizon management, progress tracking, and self-verification rather than raw step-level reasoning.

The benchmark’s limitations (small task count, single harness, no confidence intervals, static design) are real but do not undermine its core contribution. LHTB fills an important gap between the saturated short-horizon benchmarks and the completely open question of how agents perform on day-long or week-long autonomous workflows. The fact that GPT-5.5 — the best model tested — achieves only 15.2% pass@1 on tasks that average 85 minutes suggests there is substantial headroom for improvement, and that current progress estimates based on short-horizon benchmarks are likely too optimistic about real-world agent deployment readiness.

Deep Dive: What Makes Long-Horizon Tasks Fundamentally Hard

The Horizon-Length Capability Cliff

The LHTB results reveal something more interesting than “long tasks are harder.” There is a qualitative regime change at long horizons that cannot be explained by simple scaling of short-horizon difficulty. Consider:

• At short horizons (10–20 steps), agents fail mostly on hard reasoning steps: parsing a complex specification, generating the right algorithm, understanding an error message.
• At long horizons (100–500 steps), the local reasoning per step is often easier (individual shell commands are not complex), but agents fail on horizon management: maintaining correct beliefs about what is done vs. undone, prioritizing remaining sub-goals, and avoiding re-doing work.

This is the distinction between step-level intelligence and horizon-level intelligence. LHTB measures both, and the data says that frontier models have strong step-level intelligence but weak horizon-level intelligence.

Why Context Windows Are Insufficient for Long-Horizon Completion

One might expect that extending the context window is the solution: if the agent can remember all prior steps, it can avoid re-doing them. But LHTB shows this is not the binding constraint. Most models use 4–20M tokens per task (far more than current context windows allow for a single pass). Harnesses manage this with sliding windows, truncation, or summarization — but these introduce lossy compression of the agent’s prior context.

The implication is that the problem is not fundamentally about context length but about context compression quality: how well the agent can summarize its progress in a compact representation that preserves the information needed to continue making forward progress. Current agents compress their context poorly: they summarize action history but lose track of sub-goal completion status, verified assumptions, and the structure of the remaining work.

The Verification Gap at Long Horizons

A human expert completing a long-horizon task does not just execute steps; they periodically verify that prior steps achieved their intended outcomes. If step 50 produced a wrong intermediate output that was not checked, all subsequent steps building on it are wasted. Current agents run verification less rigorously: they check that a command did not produce an error, but rarely verify that the output satisfies the deeper semantic contract required by later steps.

LHTB’s hidden verifier is designed to exploit this gap: visible tests check surface properties (command ran, file exists), but hidden stress tests check deep semantic properties (numerical tolerances, schema generalization, edge case handling). Agents that pass the visible tests but not the hidden ones have done the work but not verified it deeply enough — exactly what the false-finish analysis reveals.

Pseudocode: Why Agents Fail at the 90% Mark

Algorithm 3: Typical Long-Horizon Agent Failure Pattern on LHTB

Given: task T, 90-minute budget, hidden verifier V_hidden, visible verifier V_public

Phase 1 (0–40 min): Setup and early exploration
1:  agent reads instruction
2:  agent explores directory structure
3:  agent runs visible public tests → most fail (expected, task is broken)
4:  agent identifies 3–5 visible errors to fix

Phase 2 (40–75 min): Targeted repair
5:  agent fixes the identified errors one by one
6:  after each fix: agent runs V_public to check progress
7:  eventually: V_public passes all visible tests

Phase 3 (75–90 min): Verification and stopping
8:  agent runs V_public one final time → all pass
9:  agent concludes: "task complete"
10: agent exits with ~10 minutes remaining  ← FALSE FINISH

But V_hidden contains:
- Schema variant with renamed fields (agent never saw this variant)
- Numerical tolerance check at 1% (agent only tested to 5%)
- Edge case with gzip+base64 encoding (not in visible test suite)
11: V_hidden evaluates → R = 0.80–0.92 (near-miss, not a pass)

Why this fails:
- Agent did not know V_hidden existed
- Agent did not generate adversarial self-tests
- Agent stopped when V_public said "done"

This pattern explains the 14 false-finish runs with R ≥ 0.75. The agent is not incompetent; it completed most of the task correctly. The failure is epistemological: the agent cannot verify what it cannot see.

Extended Related Work

The METR Time-Horizon Framework

A useful way to contextualize LHTB is through METR’s time-horizon analysis, which characterizes agent capability as “the typical human-time duration of tasks that the agent can complete at a fixed success rate (e.g., 50%).” Under this framing:

• Current short-horizon benchmarks (SWE-Bench, HumanEval) correspond to tasks that take humans 10–30 minutes.
• LHTB tasks correspond to tasks that take humans 2–8 hours.
• Future long-horizon benchmarks might correspond to tasks that take humans days or weeks.

The time-horizon framework predicts that agents achieve much lower success rates as horizon length grows, which LHTB validates empirically (from >60% on Terminal-Bench to ~4% mean on LHTB). What LHTB adds is the mechanism: it is not reasoning quality that degrades, but horizon management.

Process Reward Models vs. Dense Environment Graders

Process reward models (PRMs) provide dense supervision over intermediate reasoning steps in language model training. LHTB’s subtask-based grading provides dense evaluation at test time in a similar spirit. The key difference:

• PRMs use a learned model to score intermediate steps, which introduces learned bias and generalization errors.
• LHTB uses deterministic, environment-grounded graders for each subtask, which are objective and unambiguous.

This makes LHTB’s grading more reliable for evaluation, but also more expensive to construct (each task requires a carefully designed verifier). The deterministic grader design is a deliberate choice that sacrifices scalability for reliability.

SWE-Bench, SWE-Marathon, and the Progressive Horizon Stack

LHTB fits into a progressive stack of increasingly long-horizon benchmarks:

Benchmark	Typical task duration	Primary metric
HumanEval	< 1 min	Function-level correctness
SWE-Bench	10–30 min	Repository-level issue resolution
Terminal-Bench 2	< 20 min	Terminal task completion
LHTB	85+ min	Long-horizon workflow completion
SWE-Marathon	Hours	Ultra-long-horizon software work

Each step up the stack collapses current best-model pass rates from >60% to near-zero. LHTB occupies the “85+ minutes” tier, which is practically important (real professional workflows take this long) and empirically revealing (frontier models have not solved it).

Deeper Analysis: The Token Efficiency Question

How Do Models Differ in Horizon Efficiency?

Looking at the token counts in Table 1, models vary enormously in tokens-per-task:

• GPT-5.5: 4.16M tokens — low token usage, highest pass rate.
• MiniMax M3: 20.20M tokens — high token usage, 6.5% pass rate.
• DeepSeek V4 Pro: 14.45M tokens — high token usage, 6.5% pass rate.
• Hy3: 17.21M tokens — high token usage, 2.2% pass rate.

This is not a simple efficiency story. GPT-5.5 uses 5× fewer tokens than MiniMax M3 while achieving 2× better pass rate. There are at least three explanations:

1. Quality efficiency: GPT-5.5 makes more progress per token because its reasoning is better.
2. Context compression: GPT-5.5 compresses its context more aggressively, allowing it to cover more steps in the same budget.
3. Stopping behavior: Models that consume more tokens may be more verbose per step, or may re-read their entire context more often.

The paper does not disentangle these explanations, which is an important gap. Understanding why GPT-5.5 achieves high efficiency while MiniMax M3 does not would be directly actionable for improving other models.

Episode Count vs. Token Count

Episode count (number of actions taken) and token count are not proportional:

• GPT-5.5: 208 episodes, 4.16M tokens → ~20K tokens/episode
• MiniMax M3: 314 episodes, 20.20M tokens → ~64K tokens/episode
• DeepSeek V4 Pro: 321 episodes, 14.45M tokens → ~45K tokens/episode

MiniMax M3 uses 3× more tokens per episode than GPT-5.5. This suggests that MiniMax M3 is doing much more context reading (or generating longer outputs) per action, which could indicate either more thorough reasoning per step or more inefficient context management.

The Pareto Efficiency Calculation

For a model to be on the cost-reward Pareto frontier, it must satisfy:

$$\forall j \neq i: \text{cost}_j \leq \text{cost}_i \Rightarrow \text{passrate}_j \leq \text{passrate}_i \tag{8}$$

Formally, model $i$ is Pareto-dominated if there exists model $j$ with lower cost AND higher pass rate. The Pareto frontier consists of all non-dominated models.

From the reported data:

• Hy3 ($2.47, 2.2%): non-dominated (cheapest, any cheaper model would need 0% pass rate to be better)
• Doubao Seed 2.1 Pro ($5.16, 4.3%): non-dominated (better than Hy3 on pass rate at moderate cost)
• MiniMax M3 ($6.13, 6.5%): non-dominated (better pass rate than Doubao at comparable cost)
• GPT-5.5 ($21.46, 15.2%): non-dominated (highest pass rate, no model dominates it)

Models like GPT-5.4 ($27.57, 2.2%) are clearly dominated: more expensive than GPT-5.5 with much lower pass rate. This is a qualitative failure: GPT-5.4 reaches higher token counts per task precisely because it needs more attempts to make the same progress as GPT-5.5, but charges comparable prices — resulting in higher total cost for lower outcomes.

Practical Implications: Building Better Long-Horizon Agents

Implication 1 — Horizon Budgeting as a First-Class Problem

If 79% of failures are timeouts, the practical priority for agent improvement is not smarter per-step reasoning but smarter budgeting. An agent that can allocate its remaining time budget across remaining sub-goals — and stop trying to perfect an already-good enough solution — would directly improve LHTB scores. This is analogous to time management in human problem solving: knowing when to stop polishing and start submitting is as important as knowing how to solve the problem.

Concrete directions:

• Teach agents to maintain an explicit sub-goal checklist and update it after each action.
• Give agents a running estimate of remaining time budget and train them to make decisions conditioned on it.
• Penalize unnecessary re-verification loops that consume time without adding progress.

Implication 2 — Adversarial Self-Testing for Verification

The false-finish pattern implies that agents need to construct their own adversarial tests rather than relying solely on provided visible tests. A useful direction: after passing all visible tests, the agent explicitly reasons about what kinds of inputs the hidden tests might use and generates a small battery of adversarial inputs to self-test against. This reflects the reasoning of an experienced engineer who, after fixing a bug, thinks “what corner cases might still fail?” and probes them before declaring the task done.

Implication 3 — Long-Horizon Memory Architectures

Episode counts of 200–300+ are far beyond what can fit in a single context window without compression. Current harnesses use sliding window truncation, but this loses critical information about early decisions and their rationale. Better architectures might use:

• Hierarchical memory: summarize completed sub-goals into a compact state representation that is prepended to each new context window.
• Write-ahead logs: maintain a structured log of verified sub-goal states that the agent can query.
• External state tracking: store the agent’s current belief about task completion in a structured format (JSON or equivalent) rather than in unstructured text history.

Implication 4 — Calibrated Stopping Criteria

Agents currently stop when they judge the task complete based on visible signals. A better stopping criterion would include:

• Uncertainty-aware stopping: only stop when the agent’s confidence that hidden stress tests will also pass exceeds a threshold.
• Coverage-based stopping: generate a checklist of plausible failure modes and verify each one before stopping.
• Conservative default: by default, use remaining time budget to run further verification before declaring done.

Implication 5 — What LHTB Means for the “Agents Are Ready” Narrative

There is a narrative in the industry (partly fueled by impressive SWE-Bench scores) that AI coding agents are ready for semi-autonomous deployment on real engineering workflows. LHTB provides important pushback: on tasks calibrated to what a competent human expert does in 2–8 hours, the best model (GPT-5.5) succeeds 15.2% of the time. That means 84.8% of the time, a human would need to check the output, detect incompleteness, and re-attempt.

For deployment contexts where the human is reviewing every output anyway, this is acceptable — the agent is a productivity booster, not a replacement. But for contexts where the expectation is autonomous completion (running a 3-day ML training pipeline, preparing a regulatory audit), the gap between current capability (15%) and required reliability (90–99%) is not a 2× improvement problem; it is a fundamental capability gap that requires new architectures or training paradigms.

LHTB is valuable not just as a benchmark but as a calibration tool: it gives developers and deployers a realistic estimate of where agents actually stand on the difficulty spectrum that matters for real professional work.

Figure 6: Frontier Model Capability Spectrum

To put LHTB in context, here is a schematic comparison of how frontier models perform across benchmark difficulty tiers in 2026:

xychart-beta
    title "Agent pass rates across benchmark difficulty (approximate, frontier models 2026)"
    x-axis ["HumanEval", "SWE-Bench V.", "Terminal-Bench 2", "LHTB (τ=0.95)", "LHTB (τ=1.0)"]
    y-axis "Best model pass rate (%)" 0 --> 100
    bar [96, 71, 52, 15, 11]

The cliff from Terminal-Bench 2 (~52%) to LHTB (15.2%) is steep: extending the task horizon by ~4× reduces the best-model pass rate by 3.4×. This is not a smooth degradation but a qualitative transition between regime where local reasoning is sufficient and a regime where horizon management is required.

References

[1] Li et al. Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading. arXiv:2607.08964, July 2026.

[2] Yang et al. SWE-Bench: Can Language Models Resolve Real-World GitHub Issues? arXiv:2310.06770.

[3] Desai et al. SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work? arXiv:2606.07682, 2026.

[4] Li et al. Terminal-Bench / Terminus-2. Referenced as [22] in the paper.

[5] METR. Evaluating Long-Horizon Autonomy of AI Agents. METR Technical Report, 2026.

[6] DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348, 2026.