terminal-bench-loop

Drives one Terminal-Bench task through bounded smoke, diagnosis, and board-gated fix iterations.

Updated Mar 26, 2026
One-click install
npx skills add https://github.com/nabitllc/todero --skill terminal-bench-loop-nabitllc
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: terminal-bench-loop
Source: https://github.com/nabitllc/todero/tree/main/.agents/skills/terminal-bench-loop
Command: npx skills add https://github.com/nabitllc/todero --skill terminal-bench-loop-nabitllc

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Running a Terminal-Bench task repeatedly against a Todero worktree often stalls in ambiguous review states, unbounded retries, or fixes applied without approval. This Skill enforces a bounded loop with explicit issue topology, typed waiters, and board confirmation so every iteration ends in a known state with a named next-action owner. ## Core Features & Use Cases - Bounded iteration loop: Runs a single Terminal-Bench task as a smoke with an iteration budget, wall-clock caps, and explicit stop rules (pass, board rejection, budget exhausted, or named blocker). - Structured diagnosis: Applies a diagnose-why-work-stopped pattern to classify failures as task/model, Todero product, harness/setup, verifier/infrastructure, security, or unclear, with evidence quoted from run ids and status transitions. - Board-gated fixes: Requires a request_confirmation interaction on the iteration child before any implementation child is created, keeping product fixes auditable and approved. - Worktree continuity: Pins the benchmark command to an isolated Todero App worktree via PAPERCLIPAI_CMD and inheritExecutionWorkspaceFromIssueId so fixes and reruns share one workspace. - Use Case: An operator is asked to drive terminal-bench/fix-git until it passes; the Skill opens a loop parent issue, runs bounded smokes, diagnoses each stop, proposes product rules for board approval, and closes with QA and CTO review. ## Quick Start Ask the agent to loop the terminal-bench/fix-git task through Todero with an iteration budget of three until the smoke passes.

Frequently Asked Questions about terminal-bench-loop

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a Terminal-Bench task in a loop until it passes?▼

Open a top-level loop issue with the task name, iteration budget, benchmark command, and worktree pointer, then create one iteration child per run. Each iteration runs a bounded smoke, records a diagnosis, and stops on pass, board rejection, budget exhaustion, or a named blocker.

How do I diagnose why a Terminal-Bench run stopped?▼

Walk the Todero issue tree the smoke produced and find the exact issue and status that stopped progress, quoting run ids and status transitions. Classify the failure as task/model, Todero product, harness/setup, verifier/infrastructure, security, or unclear before proposing any fix.

Can this skill submit comparable Terminal-Bench benchmark results?▼

No. The loop produces smoke runs that are explicitly non-comparable by design. Comparable or full-suite benchmark submissions must be handed off to BenchmarkQualityManager and BenchmarkForensics instead of relabeling a smoke run.

When should I not use the Terminal-Bench loop skill?▼

Do not use it to build or change the todero-bench harness itself, to submit ranked benchmark results, or for ordinary Todero product bugs unrelated to a loop. Those cases follow normal engineering or investigation flows.

Why does a Harbor run create BEN-1 with no heartbeat agents?▼

That happens when the dispatch runner config is omitted, leaving the benchmark issue unassigned with zero heartbeat-enabled agents. Rerun with the full recorded command block including PAPERCLIP_HARBOR_RUNNER_CONFIG, assignee, heartbeat strategy, and agent adapter.

Can the loop apply product fixes without board approval?▼

No. Every product fix requires a request_confirmation interaction on the iteration child targeting the latest plan revision. Implementation, QA, and rerun children are only created after the board accepts the proposal.