analyzing-quality-metrics

Defines, computes, and interprets test quality metrics for QA dashboards and stakeholder reports.

113|17|Updated Oct 5, 2025
One-click install
npx skills add https://github.com/jaktestowac/awesome-copilot-for-testers --skill analyzing-quality-metrics-jaktestowac
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: analyzing-quality-metrics
Source: https://github.com/jaktestowac/awesome-copilot-for-testers/tree/main/plugins/analyzing-quality-metrics/skills/analyzing-quality-metrics
Command: npx skills add https://github.com/jaktestowac/awesome-copilot-for-testers --skill analyzing-quality-metrics-jaktestowac

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Quality numbers like coverage percentages and bug counts are routinely misused as quality gates or targets, which corrupts them and misleads stakeholders. This Skill provides precise metric definitions, baselining, counterweight pairing, and corruption checks so reported numbers survive being acted on. ## Core Features & Use Cases - Precise Metric Definitions: Formulas, data sources, time windows, exclusions, and distortions for pass rate, flake rate, escape rate, mutation score, suite duration, and more. - Anti-Metric Guardrails: A quarterly corruption checklist that detects Goodhart effects such as coverage rising while mutation score stays flat, or pass rate rising alongside skip counts. - Stakeholder Report Template: A worked quality report structure with trend tables, annotated events, data gaps, and interpretations with competing explanations. - Use Case: A QA lead asked to build a release-quality dashboard uses this Skill to define flake rate and escape rate precisely, baseline them over past releases, pair each metric with its counterweight, and present trends with caveats to engineering leadership. ## Quick Start Use the analyzing-quality-metrics skill to define and compute flake rate and escape rate from our CI history and issue tracker, then draft a stakeholder quality report for the last release.

Frequently Asked Questions about analyzing-quality-metrics

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I measure test flakiness in CI?▼

Flake rate is computed as tests with both a pass and a fail on the same commit divided by total tests, over a 14 or 30 day window. It requires CI history keyed by commit SHA, and should be reported alongside quarantine count since quarantining hides flakes.

What is defect escape rate and how is it calculated?▼

Defect escape rate is defects found in production divided by total defects found, per release, trended over 6 to 12 releases. It excludes third-party outages and incidents no test could have caught, and needs an environment-found field in the issue tracker.

Is code coverage a good quality metric?▼

Line coverage measures which lines execute during tests, not which behaviours are verified, so it is a good absence detector but a poor presence detector. Pair it with mutation score or assertion density, and never use it as a hard quality gate.

Which QA metrics should never be tracked?▼

Avoid bugs found per tester, defects introduced per developer, raw test count, coverage as a gate, automation percentage, and zero known defects. Each rewards gaming the proxy rather than improving quality, and individual-level metrics change behaviour for the worse.

How do I detect when a quality metric has been gamed?▼

Run a quarterly corruption check: coverage up with flat mutation score, pass rate up with rising skip count, flake rate down with rising quarantines, or defects down while incidents stay flat all signal corruption. Retire or redefine any metric that fires a check.