ab-testing

Designs and analyzes A/B tests and growth experimentation programs with statistical rigor.

Updated Mar 6, 2026
One-click install
npx skills add https://github.com/Transform24/THE-QUIET-AUTHORITY --skill ab-testing-transform24
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ab-testing
Source: https://github.com/Transform24/THE-QUIET-AUTHORITY/tree/main/content-ops/_factory/skills/ab-testing
Command: npx skills add https://github.com/Transform24/THE-QUIET-AUTHORITY --skill ab-testing-transform24

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Teams often run experiments without clear hypotheses, adequate sample sizes, or disciplined analysis, leading to false positives and wasted traffic. This Skill guides the full lifecycle of A/B testing—from hypothesis framing and sample size calculation to result analysis and playbook documentation—so experiments produce statistically valid, actionable outcomes. ## Core Features & Use Cases - Hypothesis & Test Design: Structures hypotheses with a proven framework, selects test types (A/B, A/B/n, MVT, split URL), and defines primary, secondary, and guardrail metrics. - Sample Size & Duration Planning: Provides quick-reference tables, duration formulas, and guidance on sequential testing and multi-variant adjustments via detailed reference guides. - Growth Experimentation Program: Supports ICE prioritization, experiment velocity tracking, cadence rituals, and a reusable experiment playbook for compounding wins. - Use Case: A marketer wants to test a new pricing page headline. The Skill calculates required sample size from baseline conversion and traffic, defines metrics, warns against peeking early, and produces a structured test plan and results documentation template. ## Quick Start Ask the assistant to help you design an A/B test for a specific page or change, providing your current conversion rate and traffic volume.

Frequently Asked Questions about ab-testing

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I calculate sample size for an A/B test?▼

Sample size depends on your baseline conversion rate, minimum detectable effect, 95% significance level, and 80% statistical power. Use the quick reference tables in the sample size guide or online calculators like Evan Miller's or Optimizely's to find the required visitors per variant.

How long should I run an A/B test?▼

Run until you reach your pre-calculated sample size, with a minimum of one full week to capture day-of-week variation. Avoid running longer than 4-8 weeks due to novelty effects and external factors, and never stop early based on peeking at results.

What is the difference between A/B testing and multivariate testing?▼

A/B testing compares two versions with a single change, while multivariate testing (MVT) tests combinations of multiple elements simultaneously. MVT requires dramatically more traffic since combinations multiply, so sequential A/B tests are often better for lower-traffic sites.

Can I stop an A/B test early if one variant is winning?▼

Stopping early based on peeked results inflates false positive rates and leads to wrong decisions. Pre-commit to your sample size and duration; if you need early-stopping flexibility, use sequential testing methods supported by tools like Optimizely or VWO.

What metrics should I track in an A/B test?▼

Define three tiers: a primary metric tied directly to your hypothesis, secondary metrics that explain why the change worked, and guardrail metrics that ensure nothing important gets worse. This structure prevents cherry-picking and protects against harmful side effects.

What should I do when my A/B test result is inconclusive?▼

An inconclusive result usually means the effect was smaller than your minimum detectable effect or traffic was insufficient. Consider running a bolder variant, increasing sample size, extending duration within limits, or making the decision with qualitative data instead.