skills.rest
agnivon
agnivonCOMMUNITY
Explore all skills by this author →
Website𝕏@agnivonGitHub

ab-testing

Designs and analyzes A/B tests with hypothesis frameworks, sample size calculations, and statistical rigor.

Updated Jun 7, 2026
Search Tags:#conversion optimization#hypothesis#experimentation#a/b testing#sample size#statistical significance#growth experiments
One-click install
npx skills add https://github.com/agnivon/viral_thread_generator --skill ab-testing-agnivon
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ab-testing
Source: https://github.com/agnivon/viral_thread_generator/tree/main/.agents/skills/ab-testing
Command: npx skills add https://github.com/agnivon/viral_thread_generator --skill ab-testing-agnivon

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Teams often run experiments without clear hypotheses, stop tests too early, or misread results, leading to false conclusions and wasted traffic. This Skill provides a structured methodology for planning, running, and analyzing A/B tests and growth experimentation programs that produce statistically valid, actionable results. ## Core Features & Use Cases - Hypothesis & Test Design: Builds structured hypotheses, selects test types (A/B, A/B/n, MVT, split URL), defines primary/secondary/guardrail metrics, and plans traffic allocation. - Statistical Planning & Analysis: Provides sample size tables, duration calculators, significance interpretation, and guidance on avoiding the peeking problem. - Experimentation Program Management: Covers ICE prioritization, experiment velocity tracking, playbooks for winning patterns, and cadence for weekly/monthly reviews. - Use Case: A product team wants to test a new pricing page headline. The Skill calculates the required sample size from their baseline conversion rate, defines metrics, warns against stopping early, and provides a results documentation template. ## Quick Start Ask the assistant to help design an A/B test for your landing page headline, providing your current conversion rate and monthly traffic.

Frequently Asked Questions about ab-testing

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I calculate sample size for an A/B test?▼

Sample size depends on your baseline conversion rate, minimum detectable effect, significance level (usually 95%), and power (usually 80%). Use the quick reference tables in the sample size guide or calculators like Evan Miller's or Optimizely's to get per-variant requirements.

How long should I run an A/B test?▼

Run until you reach the pre-calculated sample size, with a minimum of one full week to capture day-of-week variation. Avoid running longer than 4-8 weeks due to novelty effects and external factors, and never stop early based on peeking at results.

What is the difference between A/B, A/B/n, and multivariate tests?▼

A/B tests compare two versions with a single change, A/B/n tests multiple variants requiring roughly 1.5-2x more traffic, and multivariate tests combine several elements simultaneously, multiplying combinations and demanding very high traffic volumes.

Why is my A/B test showing significance after only a few days?▼

Early significance is often misleading due to the peeking problem, regression to the mean, and audience mix shifts. Checking results before reaching sample size inflates false positive rates, so commit to the full pre-calculated duration or use sequential testing methods.

What metrics should I track in an A/B test?▼

Define three tiers: a primary metric tied directly to your hypothesis, secondary metrics that explain why the change worked, and guardrail metrics that catch negative side effects. For example, a signup form test tracks completion rate, lead quality, and total signup volume.

When should I not run an A/B test?▼

Skip testing when traffic cannot reach the required sample size within a reasonable duration, when the change is too small to detect, or when the risk of a bad variant is unacceptable. Alternatives include qualitative research, testing upstream pages, or accepting a larger minimum detectable effect.

Related Skills

Explore More Marketing & Sales Skills→
aitytechaitytech
595

ab-test-setup

Designs A/B tests with hypotheses, sample size calculations, and statistical analysis plans.

Official
Intermediate
teodorboevteodorboev

ab-testing

Design, manage, and evaluate social media content experiments with statistical methods.

Community
Advanced
Veloxia-agencyVeloxia-agency

ab-test-setup

Design statistically valid A/B tests with hypotheses, metrics, and sample sizes.

Community
Advanced
WcofWcof
1

pm-interview

Generate structured user interview scripts from PMContext for product discovery.

Community
Advanced