skills.rest
thurn
thurnCOMMUNITY
Explore all skills by this author →
GitHub

adversarial-test-validation

Injects temporary user-visible bugs to verify test suites catch real regressions.

1|Updated Dec 18, 2009
Search Tags:#quality assurance#test coverage#mutation testing#test validation#regression tests#adversarial testing
One-click install
npx skills add https://github.com/thurn/dotfiles --skill adversarial-test-validation-thurn
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: adversarial-test-validation
Source: https://github.com/thurn/dotfiles/tree/main/.llms/skills/adversarial-test-validation
Command: npx skills add https://github.com/thurn/dotfiles --skill adversarial-test-validation-thurn

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Test suites often pass while missing real product regressions, leaving teams with false confidence in their coverage. This Skill deliberately introduces realistic user-visible bugs into production code, checks whether existing tests catch them, and adds durable regression tests for the gaps that survive. ## Core Features & Use Cases - Manual Mutation Testing: Introduces one behavior-level bug at a time (reversed conditions, omitted state updates, misrouted actions) and classifies it as caught or survived by the existing suite. - Regression Test Generation: For every surviving mutation, writes a black-box test at the public boundary that fails against the bug and passes after the code is restored. - Safe Restoration: Guarantees all intentional production mutations are removed, preserving pre-existing user changes and never weakening existing tests. - Use Case: Before a major release, run five adversarial iterations against a checkout flow to confirm the test suite blocks realistic failures like skipped payment state updates or miswired navigation. ## Quick Start Use the adversarial-test-validation skill to challenge this project's tests with five realistic user-visible bugs and add regression tests for any that survive.

Frequently Asked Questions about adversarial-test-validation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check if my tests catch real bugs?▼

Use manual mutation testing: introduce a small realistic bug into production code, run the ordinary test suite, and see whether any test fails for that specific reason. This Skill automates that cycle and adds regression tests for bugs the suite misses.

What is manual mutation testing without a framework?▼

Manual mutation testing means hand-editing production code to simulate a realistic defect, then running the existing test suite to see if it detects the change. Unlike automated tools, it focuses on a few high-confidence, user-visible bugs rather than thousands of mechanical mutations.

What kinds of bugs qualify as valid mutations?▼

A valid mutation must cause a failure a normal user could observe at a product boundary like the UI or CLI, be clearly wrong behavior, and be something tests should reasonably block. Syntax errors, internal-only state changes, and arbitrary data tweaks do not qualify.

Does mutation testing modify my production code permanently?▼

No. Each intentional mutation is tracked, kept active only during its iteration, and then restored exactly. The final diff contains only the new regression tests and any pre-existing user changes, never leftover mutations.

What happens when a mutation survives the test suite?▼

A surviving mutation reveals a real coverage gap. While the bug is still active, a minimal black-box regression test is added at the public boundary, verified to fail against the mutation, and then confirmed to pass once the production code is restored.

When should I not run adversarial test validation?▼

Avoid it when the baseline test suite is red or flaky, since new failures cannot be distinguished reliably, and never run it against production data or real external side effects. It also requires an isolated development environment to be safe.

Related Skills

Explore More Software Engineering Skills→
vmobifystudiovmobifystudio
4

mutation-testing

Apply behavioral mutations to validate test suite regression detection.

Community
Advanced
adityaarsharmaadityaarsharma
2

orbit-qa-regression-pack

Manage regression test suites with Playwright and CI pipelines.

Community
Advanced
Nandansai08Nandansai08
1

coverage-analysis

Analyzes code coverage reports to identify untested branches and prioritize testing effort.

Community
Intermediate
cskirocskiro
7

mutation-testing

Apply mutation testing to JavaScript, Java, and Python code bases.

Community
Advanced