Statistical Significance
A measure of confidence that a difference between two variants in a test is real rather than random chance. Expressed through a confidence level, commonly ninety-five percent, it tells you whether an A/B test result is trustworthy enough to act on before you declare a winner.
Why It Matters
Without significance, you risk shipping changes based on noise, chasing lifts that evaporate at scale. Waiting for significance protects revenue by ensuring test wins are repeatable rather than lucky short-term fluctuations.
Common Mistake
Peeking at results and stopping the test the moment it looks significant. Early peeking inflates false positives, so define sample size and duration before launch and let the test run to completion.