experiment-designer

Define a specific learning goal, then choose an experiment type that fits your constraints: beta program, cohort rollout, pilot, or fake door. Specify participants, treatment, comparison group, primary and guardrail metrics, and pre-register what action you will take for each outcome.

Use ab-test-designer when you can randomly split web or app traffic and need statistical sample size calculations. Use experiment-designer for everything else — pricing changes, feature betas, pilots, and fake doors where random assignment is not possible.

Use a fake door test to validate demand before building a feature. It measures interest signals only, not feature usability, so it sits lowest in the experimentation confidence hierarchy and should be followed by richer validation.

It requires a clear learning question and an experiment type such as beta, pilot, cohort, or fake door. Baseline metrics, constraints like timeline and risk tolerance, and context files like product.md and personas.md improve the output quality.

Betas and pilots provide medium confidence results that require interpretation, unlike randomized A/B tests. Self-selected participants can bias findings, so results should be triangulated with guardrail metrics and qualitative feedback.