skills.rest
serpro-workshop-fortaleza
serpro-workshop-fortalezaOFFICIAL
Explore all skills by this author →
GitHub

flaky-test-triage

Diagnose intermittent test failures and manage quarantine policies for unstable CI suites.

Updated Sep 10, 2026
Search Tags:#pytest#Playwright#flaky tests#CI stability#test triage#test quarantine#intermittent failure
One-click install
npx skills add https://github.com/serpro-workshop-fortaleza/datacorp-sifap-modernization-team-kit --skill flaky-test-triage-serpro-workshop-fortaleza
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: flaky-test-triage
Source: https://github.com/serpro-workshop-fortaleza/datacorp-sifap-modernization-team-kit/tree/main/.github/skills/flaky-test-triage
Command: npx skills add https://github.com/serpro-workshop-fortaleza/datacorp-sifap-modernization-team-kit --skill flaky-test-triage-serpro-workshop-fortaleza

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Intermittent test failures erode trust in CI pipelines and waste engineering time on false alarms. This Skill provides a structured workflow to reproduce flaky tests, categorize their root causes, fix them properly, and quarantine the ones that cannot be fixed quickly. ## Core Features & Use Cases - Reproduction Protocol: Run a suspect test 50+ times in isolation using Playwright's --repeat-each or pytest's --count to measure its actual flake rate. - Root Cause Taxonomy: Classify instability into five categories: async/timing issues, order dependency, external dependencies, non-determinism, and resource contention. - Quarantine Policy: Apply a flaky/ label, open a GitHub tracking issue, and enforce a 30-day SLA to fix or delete quarantined tests, with automatic quarantine above a 5% flake rate. - Use Case: Your CI fails on a test that passes on retry. Use this Skill to reproduce it 50 times, identify a missing await as the cause, replace a fixed sleep with an explicit wait, and log the resolution in the triage template. ## Quick Start Ask the AI to triage a flaky test by saying: this test fails intermittently in CI, help me reproduce it, find the root cause, and decide whether to fix or quarantine it.

Frequently Asked Questions about flaky-test-triage

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I reproduce a flaky test locally?▼

Run the test in isolation at least 50 times using Playwright's --repeat-each 50 flag or pytest --count=50. If it fails fewer than once in isolation, the flakiness likely depends on execution order or shared state with other tests.

What are the common root causes of flaky tests?▼

Flaky tests fall into five categories: async/timing issues like missing awaits or fixed sleeps, order dependency from shared state, external dependencies like network or clock, non-determinism from unordered maps or random seeds, and resource contention over ports or file locks.

When should I quarantine a flaky test instead of fixing it?▼

Quarantine a test when it cannot be fixed within one day. Apply a flaky/ label, open a GitHub tracking issue, and set a 30-day SLA to fix or delete it. Quarantined tests still run but do not fail the build.

Is using retry annotations a good way to handle flaky tests?▼

No, retry mechanisms like @Retry(3) are an antipattern that masks instability and encourages low-quality tests. The correct approach is fixing the root cause, such as replacing sleep calls with explicit waits and isolating shared state.

What flake rate should trigger automatic quarantine?▼

Track each test's flake rate over 100 runs and automatically quarantine anything exceeding 5%. Tests remaining in quarantine longer than 30 days should be deleted, since an unfixable test is worse than no test.

Related Skills

Explore More Software Engineering Skills→
Negentropy-LabyNegentropy-Laby
3

test-flakiness

Analyze CI logs and test histories to identify intermittent failures.

Official
Advanced
serpro-workshop-fortalezaserpro-workshop-fortaleza

flaky-test-triage

Diagnose intermittent test failures and manage quarantine policies for unstable CI suites.

Official
Intermediate
photonics-dhlphotonics-dhl

requesting-code-review

Automate pre-commit code verification with static scans and quality gates.

Community
Advanced
NotysotyNotysoty
9

Flaky Test Analyzer

Analyze test code and failure patterns to categorize flaky test root causes.

Community
Advanced