skills.rest
olavocarvalho
olavocarvalhoCOMMUNITY
Explore all skills by this author →
𝕏@olavovieiracomGitHub

ab-test

Analyze A/B test data with Frequentist and Bayesian significance testing.

1|Updated Jan 26, 2026
Search Tags:#data science#hypothesis testing#experimentation#statistical analysis#ab testing#online controlled experiments
One-click install
npx skills add https://github.com/olavocarvalho/data-agents --skill ab-test
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ab-test
Source: https://github.com/olavocarvalho/data-agents/tree/main/skills/ab-test
Command: npx skills add https://github.com/olavocarvalho/data-agents --skill ab-test

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires scipy, numpy, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a robust framework for designing, analyzing, and making confident decisions from A/B tests and online controlled experiments, ensuring your product changes are backed by statistical rigor.

Core Features & Use Cases

  • Experiment Design: Calculate necessary sample sizes and estimate test durations.
  • Statistical Analysis: Perform significance testing (Frequentist and Bayesian) to compare variants.
  • Validity Checks: Identify and mitigate common pitfalls like Sample Ratio Mismatch (SRM) and peeking.
  • Use Case: You've launched a new feature and want to know if it's improving conversion rates. Use this Skill to determine how much traffic you need, run the test, and get a clear statistical answer on whether the new feature is a winner.

Quick Start

Use the ab-test skill to calculate the sample size needed for an experiment with a baseline conversion rate of 5%, a minimum detectable effect of 10%, and 80% power.

Frequently Asked Questions about ab-test

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I calculate sample size and power analysis for an A/B test?▼

Calculate necessary sample sizes and estimate test durations by inputting baseline conversion rates and minimum detectable effects to achieve desired statistical power for A/B testing and online controlled experiments.

What is the difference between Frequentist and Bayesian significance testing for online controlled experiments?▼

Significance testing in online controlled experiments compares variants using Frequentist and Bayesian methods, providing statistical inference to determine if product changes yield significant improvements over baselines.

How do I check for Sample Ratio Mismatch and other validity issues in A/B testing?▼

Identify and mitigate common A/B testing pitfalls like Sample Ratio Mismatch (SRM) and peeking by running validity checks that ensure experiment data integrity and prevent false conclusions.

Can I run multiple variant testing with statistical corrections using scipy and numpy?▼

Perform multiple variant testing with statistical corrections using scipy and numpy, applying decision-making frameworks based on statistical inference and business constraints to compare several variants simultaneously.

When do I need statistical analysis for online controlled experiments?▼

Statistical analysis for online controlled experiments is needed when launching new features to determine if they improve conversion rates, requiring rigorous significance testing to validate product changes.

Related Skills

Explore More Data & Analytics Skills→
nguyenphpnguyenphp
41

ab-test-dashboard

Create and analyze A/B test dashboards with statistical significance calculations.

Community
Intermediate
WebWakaHubWebWakaHub
20

Analytics Reporter

Analyze raw business data into KPI dashboards and decision-support reports.

Official
Advanced
portermetricsampleportermetricsample
17

funnel-metrics

Analyze Google Ads funnel metrics using the SUMAS framework.

Community
Intermediate
nahisahonahisaho
3

scientific-causal-inference

Estimate causal effects from observational data using PSM, IPW, DID, and RDD.

Community
Advanced