Design a rigorous A/B testing program from scratch

Description

Your team runs occasional experiments but has no systematic approach — tests overlap, sample sizes are guessed, and results are cherry-picked. This sets up a structured experimentation program with proper hypothesis templates, statistical rigor, and a decision framework.

Design a structured A/B testing program for {{product_name}} that ensures statistical rigor and actionable results.

## Context
- Product: {{product_name}}
- Monthly active users: {{mau}}
- Current experimentation maturity: {{maturity_level}} (none / ad hoc / structured)
- Primary metric to optimize: {{primary_metric}}
- Key product areas to test: {{product_areas}}

## Step 1: Hypothesis Framework
Create a hypothesis template for every experiment:
1. "We believe that [change] for [user segment] will [expected impact] because [rationale]"
2. Primary metric: what will move if the hypothesis is correct?
3. Guardrail metrics: what must NOT degrade? (e.g., revenue, load time, support tickets)
4. Expected effect size: what's the minimum detectable difference worth acting on?

## Step 2: Statistical Foundation
1. Calculate minimum sample size per variation for:
   - 80% statistical power
   - 95% confidence level
   - Expected effect size from Step 1
2. Estimate test duration based on current traffic to {{product_areas}}
3. Define the stopping rules: when to call a test early (or not)
4. Choose the right test type: simple A/B, multivariate, or sequential testing

## Step 3: Experimentation Roadmap
1. List 10 experiment ideas across {{product_areas}}
2. Score each on: potential impact (1-5), effort (1-5), learning value (1-5)
3. Rank by ICE score and select the top 5 for this quarter
4. Create a testing calendar that avoids experiment collision

## Step 4: Results Framework
For each completed experiment:
1. State the result: statistically significant or not?
2. Practical significance: is the effect size worth shipping?
3. Segment analysis: did the change affect different user groups differently?
4. Document the learning regardless of outcome
5. Decision: ship to 100%, iterate, or kill

## Step 5: Culture & Process
- Define who can launch experiments (and who reviews them)
- Create a shared experiment log with standardized fields
- Set a monthly experiment review cadence
- Establish a "no HiPPO" rule: data wins over opinions when evidence is clear

Customize This Prompt