skills.rest
bongrealty
bongrealtyCOMMUNITY
Explore all skills by this author →
GitHub

ab-test-setup

Plan and design A/B tests with hypothesis, sample size, and metrics.

2|1|Updated Mar 13, 2026
Search Tags:#data analysis#hypothesis testing#experimentation#ab testing#product analytics#conversion rate optimization
One-click install
npx skills add https://github.com/bongrealty/skillcraft --skill ab-test-setup-bongrealty
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ab-test-setup
Source: https://github.com/bongrealty/skillcraft/tree/main/ab-test-setup
Command: npx skills add https://github.com/bongrealty/skillcraft --skill ab-test-setup-bongrealty

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps users plan, design, and implement A/B tests to scientifically measure the impact of changes on user behavior and key business metrics.

Core Features & Use Cases

  • Hypothesis Formulation: Guides users to create clear, testable hypotheses using a structured framework.
  • Test Design: Assists in defining test types (A/B, MVT), sample sizes, and duration based on traffic and desired lift.
  • Metric Selection: Recommends primary, secondary, and guardrail metrics to ensure comprehensive analysis.
  • Variant Design: Provides best practices for creating variants that are distinct enough to yield measurable results.
  • Use Case: A marketing manager wants to test a new call-to-action button copy on the landing page. This Skill will help them define the hypothesis, calculate the required sample size, choose the primary conversion metric (e.g., click-through rate), and outline the steps for implementation and analysis.

Quick Start

Use the ab-test-setup skill to help me design an A/B test for our signup page headline, aiming to increase conversion rates by 10%.

Frequently Asked Questions about ab-test-setup

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design an A/B test to measure changes in conversion rate optimization?▼

Design an A/B test by formulating a structured hypothesis, selecting primary and guardrail metrics, calculating the required sample size, and defining distinct variants to scientifically measure user behavior changes.

What is the difference between A/B/n tests and Multivariate Tests (MVT)?▼

A/B/n tests compare multiple distinct variants against a control simultaneously, while Multivariate Tests (MVT) evaluate combinations of multiple elements on a page to identify the optimal configuration for user flow optimization.

How do I calculate sample size and test duration for product analytics experiments?▼

Calculate sample size and test duration based on your website traffic, desired minimum detectable effect, and statistical power requirements to ensure your product analytics experiments reach significance.

What metrics should I track when running web page experimentation?▼

Track primary metrics to measure the main goal, secondary metrics for additional insights, and guardrail metrics to prevent negative impacts on other key business areas during web page experimentation.

Do I need statistical significance to evaluate hypothesis testing results?▼

Yes, statistical significance is required to confidently evaluate hypothesis testing results, ensuring observed changes in user behavior are caused by your variants rather than random variation in the data.

Related Skills

Explore More Product & Management Skills→
ScaleAcademyScaleAcademy
2

brand-health

Analyze Meta Ads account data to establish brand campaign performance baselines.

Official
Intermediate
w95w95
145

statistical-analysis

Apply descriptive statistics, trend analysis, outlier detection, and hypothesis testing to datasets.

Community
Intermediate
phazurlabsphazurlabs
35

UX Metrics & Measurement

Define and operationalize UX measurement frameworks like HEART, SUS, and UEQ.

Community
Advanced
brainbytes-devbrainbytes-dev
3

ad-creative-testing

Plan A/B, multivariate, and champion/challenger tests for ad campaigns.

Community
Advanced