ai-testing-strategy

Define a testing scope matrix covering six test types (functional, performance, security, compliance, fairness, integration) across six system layers, then add model-specific tests for accuracy, adversarial robustness, drift, and regression. This Skill generates that full strategy document with measurable thresholds.

AI systems need data quality tests (schema, distribution, lineage), model tests (accuracy, calibration, adversarial, counterfactual), fairness tests (demographic parity, disparate impact), and continuous production monitoring for drift. Traditional unit tests alone miss data-driven failures.

Measure demographic parity, equal opportunity, and disparate impact ratio (four-fifths rule) across protected groups, plus intersectional analysis for attribute combinations. The Skill defines thresholds such as 90% parity minimum and includes explainability and audit trail verification.

No. It produces the testing strategy, decision tables, and gate definitions, not executable pytest fixtures or implementation code. For test implementation, use a testing framework skill; for non-AI software testing, use a general testing-strategy skill.

Use proxy metrics such as human evaluation, downstream task performance, and A/B testing against business outcomes. The strategy also covers uncertainty limits and delayed-outcome validation for unsupervised or generative systems.

Bottom-up suits data-heavy batch systems where data quality is the main risk, top-down fits user-facing real-time systems, and parallel integration with contracts works for large multi-team platforms. Regulated systems typically combine bottom-up with a digital-twin test harness.