ab-test-setup

Designs statistically valid A/B tests with hypotheses, sample sizes, and analysis frameworks.

Updated Jul 30, 2026
One-click install
npx skills add https://github.com/executiveusa/SYNTHIA-3.1 --skill ab-test-setup-executiveusa
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ab-test-setup
Source: https://github.com/executiveusa/SYNTHIA-3.1/tree/main/03-brain/apps/control-room/openclaw-logic/synthia-3.0-backend/skills.md/marketingskills-main/marketingskills-main/skills/ab-test-setup
Command: npx skills add https://github.com/executiveusa/SYNTHIA-3.1 --skill ab-test-setup-executiveusa

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Teams often run A/B tests without proper hypotheses, adequate sample sizes, or disciplined analysis, leading to false positives and wasted traffic. This Skill guides the full experimentation lifecycle so tests produce statistically valid, actionable results. ## Core Features & Use Cases - Hypothesis Structuring: Builds testable hypotheses using a structured framework tied to observations, expected outcomes, and measurable metrics. - Sample Size & Duration Planning: Provides quick-reference tables, duration formulas, and guidance for multiple variants and low-traffic scenarios. - Test Documentation Templates: Supplies ready-to-use templates for test plans, results documentation, stakeholder updates, and prioritization scorecards. - Use Case: A growth marketer wants to test a new pricing page headline. The Skill helps define the hypothesis, calculate that 8,700 visitors per variant are needed to detect a 20% lift at a 10% baseline, set guardrail metrics, and document the final decision. ## Quick Start Help me design an A/B test for my signup page headline, including the hypothesis, sample size, and success metrics.

Frequently Asked Questions about ab-test-setup

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I calculate sample size for an A/B test?▼

Sample size depends on your baseline conversion rate, minimum detectable effect, significance level (usually 95%), and power (usually 80%). Use reference tables or calculators like Evan Miller's or Optimizely's to find the required visitors per variant.

How long should I run an A/B test?▼

Run tests until you reach the pre-calculated sample size, with a minimum of one full week to capture day-of-week variation. Avoid running longer than 4-8 weeks due to novelty effects and external factors.

What is the difference between A/B, A/B/n, and multivariate tests?▼

A/B tests compare two versions with a single change, A/B/n tests multiple variants requiring roughly 1.5-2x more sample, and multivariate tests combine multiple changes simultaneously requiring very high traffic volumes.

Why is peeking at A/B test results early a problem?▼

Checking results before reaching the target sample size and stopping early inflates false positive rates, leading to wrong decisions. Pre-commit to a sample size, or use sequential testing methods offered by tools like Optimizely or PostHog.

What should I do when my traffic is too low for an A/B test?▼

Increase the minimum detectable effect, reduce the number of variants, test higher-traffic pages earlier in the funnel, or extend the test duration. If none work, rely on qualitative data instead of running an underpowered test.