experiment-design

Design controlled experiments with hypotheses, confound hunting, randomization, and pre-registration to establish causation.

1|Updated Jul 3, 2026
One-click install
npx skills add https://github.com/Nandansai08/skillz --skill experiment-design-nandansai08
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: experiment-design
Source: https://github.com/Nandansai08/skillz/tree/main/skills/research/experiment-design
Command: npx skills add https://github.com/Nandansai08/skillz --skill experiment-design-nandansai08

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Teams routinely make causal claims from weak evidence — opt-in pilots, naked before/after comparisons, and metrics chosen after the data arrives. This Skill provides a rigorous workflow for designing experiments that actually isolate cause and effect, so decisions about process changes, tooling rollouts, and product mechanics rest on valid evidence rather than noise. ## Core Features & Use Cases - Falsifiable Hypothesis Framing: State hypotheses with a mechanism and secondary predictions, so you can distinguish "it worked" from "something worked". - Comparison Design & Confound Hunting: Choose the strongest feasible design (randomized control, staggered rollout, matched comparison, diff-in-diff) and walk a confound gallery covering selection effects, seasonality, novelty, instrumentation drift, and contamination before running. - Pre-Registration & Decision Rules: Lock the primary outcome, decision rule, guardrails, detectable effect size, and stop date before data collection to prevent post-hoc rationalization. - Use Case: A platform team wants to know whether AI-assisted code review reduces escaped defects. The Skill guides them to randomize at the team level, pre-register the defect-rate decision rule, blind the graders, and check the mechanism (pre-merge findings per PR) before adopting. ## Quick Start Ask the agent to design an experiment testing whether a specific change causes a measurable outcome, providing your constraints on randomization, units, and the smallest effect worth detecting.

Frequently Asked Questions about experiment-design

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I design an experiment to prove a change causes an outcome?▼

State a falsifiable hypothesis with a mechanism, choose the strongest feasible comparison (randomized control is the gold standard), hunt confounds before running, and pre-register one primary outcome with a decision rule. Analyze against the pre-registration and report the mechanism checks and limitations.

What is the difference between experiment design and A/B test analysis?▼

Experiment design is the general discipline for establishing causation in settings without clean traffic splits, such as team-level pilots and process changes. A/B test analysis covers the high-volume online machinery like sample ratio mismatch, peeking corrections, and statistical sizing.

When should I not use a pre/post comparison?▼

Avoid naked pre/post comparisons whenever anything else changed during the window, such as a quarter boundary, reorg, or seasonal shift. Use concurrent controls or difference-in-differences instead; pre/post is only defensible for huge, immediate effects with nothing else changing.

Why do opt-in pilot results fail when rolled out broadly?▼

Opt-in pilots suffer selection bias: volunteer teams are already more motivated, so the improvement reflects who signed up rather than the treatment. Randomize assignment or use staggered rollouts so the comparison isolates the change itself.

How many teams do I need for a pilot experiment?▼

Small unit counts mean only large effects are detectable, so state the detectable effect size upfront, such as "this pilot detects a 30% improvement, not 5%." Run complete natural cycles and pre-commit a stop date rather than extending when trends look promising.