flaky-test-diagnosis

Diagnoses intermittent test failures by reproducing flake rates and isolating root causes.

1|Updated Jul 3, 2026
One-click install
npx skills add https://github.com/Nandansai08/skillz --skill flaky-test-diagnosis-nandansai08
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: flaky-test-diagnosis
Source: https://github.com/Nandansai08/skillz/tree/main/skills/testing-qa/flaky-test-diagnosis
Command: npx skills add https://github.com/Nandansai08/skillz --skill flaky-test-diagnosis-nandansai08

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Tests that pass sometimes and fail sometimes — locally vs CI, alone vs in suite — hide real causes behind retry-until-green band-aids, tripling CI cost and occasionally masking genuine races in production code. This Skill provides a structured reproduction loop that identifies the exact root cause instead of papering over it. ## Core Features & Use Cases - Reproduction-First Workflow: Measure the actual flake rate with 200+ iteration loops (pytest-repeat, go test -count) before and after any fix, so fixes are proven rather than assumed. - Root-Cause Taxonomy: Systematically classify flakes into interference (polluting tests), time, async ordering, unordered collections, randomness, or CI environment deltas — with concrete fixes for each. - Polluter Bisection: For suite-only failures, replay the CI test order and bisect preceding tests to find the test leaking shared state, then fix the polluter rather than patching the victim. - Use Case: A test fails roughly 1 in 30 CI runs but never locally. You enable local parallelism matching CI, reproduce the flake, trace it to a check-then-insert race on a shared sequence, and ship a unique constraint plus upsert — fixing a production bug the flaky test had been detecting all along. ## Quick Start Diagnose why my test passes locally but fails intermittently in CI and identify the root cause instead of adding a retry.

Frequently Asked Questions about flaky-test-diagnosis

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I find the root cause of a flaky test?▼

First reproduce the flake rate by running the test in a loop of 200+ iterations. If it fails alone, the cause is nondeterminism inside the test; if only in suite, bisect preceding tests to find a polluting test; if only in CI, diff the environments for CPU, timezone, or parallelism differences.

How to reproduce a test that only fails in CI?▼

Imitate CI conditions locally: limit CPUs with docker run --cpus=1, set TZ=UTC, and enable the same parallelism flags CI uses. The most frequent causes are fewer CPUs triggering timeouts, different timezone or locale, and CI-only parallel execution.

Why does my test pass alone but fail when run with other tests?▼

This indicates inter-test interference: a preceding test leaks shared state such as database rows, module-level caches, mutated globals, or unpatched mocks. Replay the CI test order and bisect the preceding tests to find the polluter, then fix that test rather than the victim.

Should I add retries to fix flaky tests in CI?▼

No. Retries hide the root cause, inflate CI duration roughly threefold, and can mask real races in production code. The correct approach is measuring the flake rate, fixing the named root cause, and proving the fix with 200+ green iterations.

When should I not use flaky test diagnosis?▼

Do not use it for consistently failing tests, which are ordinary bugs requiring direct debugging. It also does not cover suite-wide e2e flakiness policy such as quarantine rules and retry budgets, which belongs to e2e-test-triage.