ab-test-setup

Design statistically valid A/B tests with hypotheses, sample sizes, and analysis checklists.

Updated Jul 28, 2026
One-click install
npx skills add https://github.com/christian-byrne/comfy-skills --skill ab-test-setup-christian-byrne
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ab-test-setup
Source: https://github.com/christian-byrne/comfy-skills/tree/main/skills/ab-test-setup
Command: npx skills add https://github.com/christian-byrne/comfy-skills --skill ab-test-setup-christian-byrne

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Teams often run A/B tests without clear hypotheses, adequate sample sizes, or statistical discipline, leading to false positives, inconclusive results, and wasted traffic. This Skill guides the design and analysis of experiments so results are statistically valid and actionable. ## Core Features & Use Cases - Hypothesis and Variant Design: Structures test hypotheses with a proven framework and helps isolate single variables across copy, design, CTA, and content changes. - Sample Size and Significance Guidance: Provides quick-reference sample size tables, calculator links, and rules for avoiding early peeking and premature test stops. - Experimentation Program Management: Covers ICE prioritization, experiment velocity metrics, playbook documentation, and weekly-to-quarterly review cadences. - Use Case: A growth marketer wants to test a new pricing page headline. The Skill helps define the hypothesis, calculate that 12k visitors per variant are needed to detect a 20% lift on a 10% baseline, set guardrail metrics, and document the result in an experiment playbook. ## Quick Start Help me design an A/B test for my signup page headline, including the hypothesis, required sample size, and how to know when the results are statistically significant.

Frequently Asked Questions about ab-test-setup

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I calculate sample size for an A/B test?▼

Sample size depends on your baseline conversion rate and the minimum detectable effect. For example, a 3% baseline detecting a 20% lift needs about 12k visitors per variant. Use calculators like Evan Miller's or Optimizely's for precise figures.

How do I write a good A/B test hypothesis?▼

A strong hypothesis follows the structure: because of an observation, we believe a change will cause an expected outcome for an audience, measured by specific metrics. Avoid vague predictions like testing a button color without reasoning or data behind it.

Why is peeking at A/B test results early a problem?▼

Checking results before reaching the pre-calculated sample size and stopping early inflates false positive rates, leading to wrong decisions. Pre-commit to your sample size and duration, and only monitor for technical issues and guardrail violations during the test.

What is the difference between client-side and server-side A/B testing?▼

Client-side testing modifies the page with JavaScript after load, which is quick to implement but can cause flicker; tools include PostHog, Optimizely, and VWO. Server-side testing determines the variant before render, eliminating flicker but requiring development work.

What should I do when an A/B test shows no significant difference?▼

An inconclusive result usually means you need more traffic or a bolder change. Check whether you reached the required sample size, review segment differences like mobile versus desktop, and consider testing a larger, more meaningful variation.