agent-testability

Audit whether a codebase lets a coding agent verify its own changes and propose fixes.

1|Updated Mar 26, 2026
One-click install
npx skills add https://github.com/arndvs/ctrlshft --skill agent-testability-arndvs
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: agent-testability
Source: https://github.com/arndvs/ctrlshft/tree/main/skills/agent-testability
Command: npx skills add https://github.com/arndvs/ctrlshft --skill agent-testability-arndvs

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Coding agents report success even when their changes are broken, because nothing in the repo contradicts them. This Skill audits whether a repository gives an agent a reliable way to verify its own work — a fast, decisive, specific feedback loop — and identifies exactly where that loop is open. ## Core Features & Use Cases - Hands-on loop verification: Make a small change, then deliberately break something to test whether the suite actually catches regressions, rather than judging tests by reading them. - Structured finding taxonomy: Classifies issues as Blocking (no runnable verification, green-when-broken suites, flaky tests, exit-code swallowing), Worth fixing (slow fast path, untestable seams, missing fixtures), or Optional. - Actionable reporting: Every finding includes a file location, what goes unverified, and a concrete fix, sequenced by loop-closing value rather than coverage gain. - Use Case: Before enabling autonomous or unattended agent runs on a repository, audit whether the agent can tell — without a human — whether each change it makes is correct, and fix the gaps first. ## Quick Start Ask the agent to audit this repository's test setup for agent usability and report what an agent cannot verify on its own.

Frequently Asked Questions about agent-testability

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I make my repo safe for autonomous coding agents?▼

Ensure an agent can run one documented command that exits zero on success and non-zero on failure, with failure output naming what broke. Audit by deliberately breaking code and checking the suite catches it, then fix gaps on the paths agents actually touch.

Why does my coding agent say it's done when the change is broken?▼

The agent finished the edit and nothing contradicted it, so it reported success. This happens when no verification command exists, the suite passes despite broken code, or failures don't identify what broke — the fix is closing that feedback loop.

What makes a test suite usable by an AI agent?▼

Three properties: discoverable (one documented command), decisive (exit status matches reality), and specific (failures name the broken behavior with expected and actual values). A fast tier under two minutes matters because agents verify repeatedly within a task.

How do I detect tests that pass when code is broken?▼

Break something on purpose — delete a branch or change a function's behavior — and see if the suite stays green. Common causes are assertions that only check a function returns, mocks that stub the logic under test, and snapshots regenerated on failure.

Why are flaky tests worse for AI agents than for humans?▼

An agent has no memory of a test being flaky last week, so it treats a spurious failure as real, investigates, and often modifies working code. Quarantine known flaky tests rather than leaving them in the main verification path.

Does high test coverage mean an agent can verify its work?▼

No. A repo with excellent coverage can still fail if the suite takes forty minutes, failures print nothing useful, or nobody can tell which command to run. Judge by whether one common class of change becomes verifiable, not by coverage percentage.