flaky-test-fixer

Diagnoses and fixes flaky tests using a 9-step forensic workflow and 20-category flakiness taxonomy.

Updated Jul 28, 2026
One-click install
npx skills add https://github.com/christian-byrne/comfy-skills --skill flaky-test-fixer-christian-byrne
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: flaky-test-fixer
Source: https://github.com/christian-byrne/comfy-skills/tree/main/skills/flaky-test-fixer
Command: npx skills add https://github.com/christian-byrne/comfy-skills --skill flaky-test-fixer-christian-byrne

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Intermittent test failures erode trust in CI pipelines and waste engineering time on re-runs. This Skill provides a structured forensic workflow to identify, reproduce, root-cause, and permanently fix flaky tests instead of masking them with retries or skips. ## Core Features & Use Cases - 9-Step Forensic Workflow: Guides you from identifying the flake in CI, downloading failure logs with the GitHub CLI, classifying the failure, reproducing it locally, through to fixing, verifying, and documenting. - 20-Category Flakiness Taxonomy: Maps error signals (timeouts, EADDRINUSE, ordering failures, timezone drift) to root-cause categories and their common fixes. - Sibling Sweep: Searches the test suite for other instances of the same antipattern (raw setTimeout, unrestored env vars, missing afterEach cleanup) so the fix is applied everywhere. - Use Case: A test passes locally but fails intermittently in CI. The Skill pulls the failed run logs, classifies the failure as a shared-state issue, reproduces it with repeated runs, fixes the root cause, and verifies with 20 consecutive passing runs. ## Quick Start Ask the agent to investigate and fix the intermittently failing test in your CI pipeline using the flaky test workflow.

Frequently Asked Questions about flaky-test-fixer

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I fix a flaky test that fails intermittently in CI?▼

Download the failed CI run logs with gh run view, classify the error against the flakiness taxonomy, reproduce it locally with repeated runs, then fix the root cause rather than adding retries. Verify with at least 20 consecutive passing runs.

How to reproduce a flaky test locally that only fails in CI?▼

Run the test repeatedly with npx vitest run --repeat 10, match CI parallelism with --maxWorkers, run the full suite to catch ordering issues, and add CPU pressure with stress-ng. If it still won't reproduce, use the CI logs as your reproduction evidence.

What are the most common causes of flaky tests?▼

The taxonomy covers 20 categories including timing races, shared mutable state, test ordering dependencies, unmocked external services, unseeded randomness, port conflicts, filesystem races, and environment variable pollution. Error messages like EADDRINUSE or varying assertion values map to specific categories.

Why does my test pass alone but fail when run with the full suite?▼

This pattern indicates shared mutable state, test ordering dependencies, memory pressure, or mock leakage between tests. Check for module-level state, missing afterEach cleanup, and unrestored mocks, then isolate state per test.

Should I add retries or skip a flaky test as a fix?▼

No. Skipping, deleting, or retrying a flaky test masks the underlying bug, which the taxonomy lists as the 'retry mask' antipattern. The workflow requires fixing the root cause and verifying stability with repeated runs.