test-flakiness

Detects flaky tests by analyzing CI logs and test result history across runs.

Updated Aug 2, 2026
One-click install
npx skills add https://github.com/claretagrapelike32/codex-ai-game-studio --skill test-flakiness-claretagrapelike32
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: test-flakiness
Source: https://github.com/claretagrapelike32/codex-ai-game-studio/tree/main/plugins/ai-game-studio/skills/test-flakiness
Command: npx skills add https://github.com/claretagrapelike32/codex-ai-game-studio --skill test-flakiness-claretagrapelike32

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Flaky tests that pass and fail intermittently erode trust in CI pipelines, causing teams to ignore genuine failures. This Skill identifies non-deterministic tests from CI run logs, classifies their likely root causes, and recommends whether to quarantine or fix each one. ## Core Features & Use Cases - CI Log Parsing: Reads JUnit/NUnit XML results and plain-text logs from Godot (GdUnit4), Unity, and Unreal test runners to build per-test pass/fail history. - Flakiness Classification: Categorizes tests by failure rate (high, moderate, suspected) and diagnoses likely causes such as timing issues, order dependency, random seeds, or floating-point comparisons. - Quarantine Management: Updates the quarantine section of tests/regression-suite.md and optionally writes a full flakiness report to production/qa/. - Use Case: After several CI runs, a team notices intermittent red builds. Run this Skill to scan test-results/ logs, identify that a physics test fails 30% of the time due to a float comparison, and get a concrete fix recommendation using epsilon comparison. ## Quick Start Ask the AI to run the test-flakiness skill in scan mode to analyze all available CI logs and report which tests are flaky with recommended actions.

Frequently Asked Questions about test-flakiness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I detect flaky tests from CI logs?▼

Run the skill in scan mode to parse all available CI logs in .github/ or test-results/ directories. It builds a pass/fail history per test across runs and flags any test showing both outcomes without code changes as flaky.

What test result formats does flaky test detection support?▼

It supports JUnit XML output from Godot GdUnit4 and Unity game-ci test runners, plus plain-text logs from Unreal automation using Result: Success/Fail patterns. You can also pass a specific log file path as an argument.

When should I quarantine a flaky test instead of fixing it?▼

Quarantine tests failing in more than 25% of runs immediately using engine-specific skip annotations. Tests failing 5-25% should be investigated and fixed directly, while tests under 5% failure rate should only be monitored as suspected flaky.

Why does my test fail intermittently with no code changes?▼

Common causes include timing/async issues, test order dependency, unseeded randomness, resource leaks, external state, and floating-point equality comparisons. The skill inspects the test file for these patterns to classify the likely cause and suggest a fix.

How many CI runs are needed for reliable flakiness detection?▼

At least 3 runs are needed for meaningful signal, and 5 or more runs provide better statistical confidence. With fewer than 3 runs, findings are flagged as suspected rather than confirmed flaky.