Three Attempts, Still Unsolved: Challenging and Verifiable Tasks in AB-Terminal Bench
Yueyang Pan,MLE
AB-Terminal Bench now includes over 2,000 new terminal-agent tasks, tested across three frontier agent stacks — Terminus-2 + Gemini 3.8 Flash, Claude Code + Claude Fable 5.1, and Codex + GPT-5.6 Sol. Even the best-performing stack failed on 43% of a random sample in a single attempt, showing that task difficulty can be engineered to keep pace with model capability while remaining solvable and verifiable.
The Frontier Has Moved — and So Has the Data
Earlier this year, we shared how we build verifier-backed terminal tasks and showed that fewer than 2,000 carefully filtered tasks could substantially improve Qwen3-32B on Terminal-Bench 2.0. For more details on our task construction pipeline and earlier post-training experiments, see our previous [AB-Terminal Bench post].
Since then, the capability frontier has continued to move. As coding agents become stronger, tasks that were difficult only a few months ago can quickly become saturated. This raises a natural question: can we continue producing tasks that remain useful for training and evaluation as frontier agents become stronger?
We have since produced over 2,000 new high-quality, high-difficulty terminal bench tasks, spanning both science-focused and general terminal workloads. Under repeated evaluation, a substantial portion remains difficult even for frontier agent stacks, with many tasks remaining unsolved in all three attempts.
Figure 1: Domain distribution of the full task collection
What “Hard” Means for Frontier Agents
We measure task difficulty empirically: how reliably can an agent solve the task under executable verification? Each task is evaluated over three independent attempts, with outcomes reported as 0/3 to 3/3 successful attempts.
We first evaluated a large subset of the collection using Terminus-2 + Gemini 3.8 Flash. Among the general terminal tasks, 54.8% were unsolved across all three attempts (0/3), compared with 38.0% of the science-focused tasks. Only 16.0% and 18.0%, respectively, were solved consistently across all three attempts (3/3).
Figure 2. Large-scale distribution of successful runs across three attempts under Terminus-2 + Gemini 3.8 Flash
Difficulty measured against a single agent stack may reflect model- or harness-specific weaknesses. To test whether these tasks remain challenging for other agent stacks, we randomly sampled 65 tasks and evaluated each with a single attempt (Pass@1) across three frontier agent stacks:
Terminus-2 + Gemini 3.8 Flash
Claude Code + Claude Fable 5.1
Codex + GPT-5.6 Sol
Figure 3. Pass@1 comparison across three frontier agent stacks on the same 65 randomly sampled tasks
On the same 65-task sample, Pass@1 ranged from 33.8% for Terminus-2 + Gemini 3.8 Flash to 52.3% for Claude Code + Claude Fable 5.1 and 56.9% for Codex + GPT-5.6 Sol. Even the highest-scoring stack in this sample failed to solve 28 of the 65 tasks in its single attempt.
These results compare complete agent stacks, not base models in isolation. The large-scale evaluation characterizes the overall difficulty distribution using repeated attempts, while the cross-agent comparison provides a controlled check on whether that difficulty persists across different frontier systems.
Where Does the Difficulty Come From?
One observation stands out:
Long-horizon does not necessarily mean hard.
One task in the collection takes more than three hours to execute, yet frontier agents solve it at Pass@3 = 3/3. Meanwhile, substantially shorter tasks remain at 0/3. Runtime, trajectory length, and environment size turn out to be weak proxies for real difficulty. The harder tasks instead combine several sources of complexity:
Multi-stage reasoning. Early decisions change later environment states, so the agent must maintain a coherent plan across dependent steps rather than solve each one independently.
Domain-specific reasoning. Especially in science-focused tasks, command-line execution must be combined with scientific or numerical reasoning, specialized libraries, or domain-specific data formats.
Debugging and recovery. Initial approaches often fail. Solving the task requires diagnosing what went wrong and revising strategy without losing track of the goal.
Interacting constraints. Functional behavior, output format, environment state, and numerical thresholds may all need to hold at once.
These sources tend to compound — difficulty comes less from any single dimension than from having to satisfy several of them together.
Figure 4: Large-scale AB-Terminal Bench Tasks: Average Runtime vs. Average Reward
Hard Does Not Mean Unsolvable
A task is not high quality because frontier agents fail it. It is high quality when frontier agents fail it despite the task being well-specified, reproducible, and verifiably solvable.
Broken environments, hidden requirements, or verifier loopholes can make a task look difficult without actually measuring agent capability. This is why every task passes validation across five areas before its pass rate is treated as evidence of difficulty:
Instruction–verifier alignment. Every substantive criterion the verifier enforces — thresholds, required artifacts, functional behavior — must appear clearly in the task instruction, not as a hidden requirement known only to the verifier.
Oracle solvability. The golden solution must execute successfully in the target environment and pass the same verifier used to evaluate agent solutions. This establishes that the task is genuinely solvable.
Negative controls and oracle integrity. A no-operation (NOP) run must not pass. Separately, reference-solution review checks for hard-coded answers, hidden-test access, and verifier shortcuts.
Environment and reproducibility. Dependencies, services, and permissions are checked for stable execution, and repeated verification must produce consistent results.
Human verification. Flagged ambiguities and suspected shortcuts go to human review; fixes must pass validation again before the task is used for difficulty measurement.
Only validated tasks move on to difficulty calibration, where each is run across frontier-agent stacks under matched conditions and predefined budgets.
Figure 5: AB-Terminal Bench task quality and difficulty validation pipeline
Why Frontier-Hard Tasks Matter
For post-training, frontier-hard tasks concentrate learning signal where capability gaps remain. Executable verifiers provide objective rewards for reinforcement learning, and both successful and failed trajectories can support supervised training and recovery-oriented learning.
Our earlier experiments showed the same difficulty gradient: after SFT, Qwen3-32B improved substantially on Easy and Medium tasks (5% → 56% and 4% → 34%, respectively), while Hard tasks saw far more modest gains (2% → 13%) and remained largely unsaturated.
For evaluation, harder tasks preserve discriminative power as existing benchmarks become easier, helping distinguish increasingly capable agent stacks and reveal remaining differences in reasoning, tool use, environment interaction, and recovery.
The goal is not difficulty for its own sake, but tasks that remain challenging while still being solvable and objectively verifiable.
Closing
Terminal-agent capability is advancing quickly, and useful task distributions have to keep pace.
The challenge is no longer simply to generate plausible terminal instructions, or even executable tasks. It is to continuously produce tasks that are difficult, diverse, solvable, verifiable, reproducible, and scalable.
As the model frontier moves, the data frontier has to move with it.
