Power Analysis
Calculation of the necessary sample size to detect an effect of a given size with desired probability (power).
Power Analysis calculates sample size BEFORE the test – without it, A/B tests are either too small (miss the effect) or too long (waste traffic).
Explanation
Power = P(detect effect | effect exists). Standard: 80% power. Four linked variables: Sample size, effect size, significance level (α), power (1-β). Fix three, calculate the fourth.
Marketing Relevance
Without power analysis, you waste traffic on tests too small (underpowered) or too long (oversized). Critical for A/B test planning.
Common Pitfalls
Setting MDE too optimistically ("2% lift is enough"). Forgetting power for subgroups. Not accounting for multiple testing.
Origin & History
Neyman & Pearson laid the foundations in the 1930s. Cohen (1969) made power analysis practical. Today tools like Evan Miller's Calculator and statsmodels provide automatic calculation.
Comparisons & Differences
Power Analysis vs. Bayesian Sample Size
Frequentist power analysis plans for α and β; Bayesian methods plan for expected posterior precision.
Power Analysis vs. Sequential Testing
Power analysis plans fixed sample size; Sequential testing allows earlier stops with statistical control.