test-flakiness

Detect flaky tests by analyzing CI logs and test result history across runs.

13|Updated Jul 30, 2026
One-click install
npx skills add https://github.com/frabcd/codex-ai-game-studio --skill test-flakiness-frabcd
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: test-flakiness
Source: https://github.com/frabcd/codex-ai-game-studio/tree/main/plugins/ai-game-studio/skills/test-flakiness
Command: npx skills add https://github.com/frabcd/codex-ai-game-studio --skill test-flakiness-frabcd

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Flaky tests that intermittently pass and fail erode trust in CI pipelines, causing teams to ignore genuine failures. This Skill identifies non-deterministic tests from run history, classifies likely root causes, and recommends whether to quarantine or fix each one. ## Core Features & Use Cases - CI Log Parsing: Reads JUnit XML results (GdUnit4, Unity game-ci) and plain-text logs from Godot, Unity, and Unreal test runners to build per-test pass/fail history. - Flakiness Classification: Categorizes tests by failure rate (high, moderate, suspected) and diagnoses likely causes such as timing/async issues, order dependency, random seeds, resource leaks, and floating-point comparisons. - Quarantine Management: Updates the quarantine section of tests/regression-suite.md and optionally writes a full flakiness report to production/qa/, always with explicit user approval. - Use Case: After several CI runs where developers dismiss failures as "probably flaky", run this Skill to statistically confirm which tests are non-deterministic and get engine-specific fix guidance. ## Quick Start Ask the AI to run the test-flakiness skill in scan mode to analyze all available CI logs and report which tests are flaky with recommended actions.

Frequently Asked Questions about test-flakiness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I detect flaky tests from CI logs?▼

Run the skill in scan mode to parse all available CI logs in .github/ or test-results/ directories. It builds a pass/fail history per test across runs and flags any test showing both outcomes without code changes as flaky.

How to find flaky tests in Godot or Unity projects?▼

For Godot, the skill reads GdUnit4 JUnit XML output from test-results/. For Unity, it parses NUnit XML from game-ci test runners. Unreal automation logs in Saved/Logs/ are scanned for Result: Success and Result: Fail patterns.

What failure rate makes a test flaky enough to quarantine?▼

Tests failing in more than 25% of runs are high flakiness and should be quarantined immediately. Tests failing 5-25% should be investigated and fixed, while 1-5% failure rates are marked as suspected and monitored.

Does this skill delete or modify my test files?▼

No, the skill never deletes test files. Quarantine means adding skip annotations and listing tests in the regression suite registry. All writes to regression-suite.md or report files require explicit user approval first.

Why does flakiness detection need multiple CI runs?▼

Statistical confidence requires run history; with fewer than 3 runs, findings are flagged as suspected rather than confirmed. A single pass/fail pair cannot distinguish genuine flakiness from a one-off environment issue.