Statistical Significance
Also known as: p-value, statistical testing, significance threshold, null hypothesis, alpha level
Statistical significance is a mathematical determination that an observed result is unlikely to have occurred by random chance.
When researchers run experiments (A/B tests, clinical trials, surveys), they measure whether the observed difference between groups is large enough that it probably reflects a real effect rather than random variation. The conventional threshold is a p-value (probability value) of 0.05: meaning there is less than a 5% chance of seeing a result this extreme if there were actually no true effect. 'Statistically significant' does not mean 'important' or 'large': a result can be statistically significant but trivially small in practical terms.
Statistical Significance, explained at three levels
At home
Statistical significance answers the question: is this result real, or could it have happened by luck? When scientists compare two groups (people who took a drug vs. people who took a placebo), they need to know whether any difference between the groups reflects a real drug effect or just natural random variation. A result is 'statistically significant' when the math says there is less than a 5% chance the observed difference happened by luck. Think of it as a bar of evidence: results have to clear this bar before researchers conclude they found something real.
At work
In business, statistical significance comes up most often in A/B testing: testing two versions of a webpage, email, or feature to see which performs better. If 10% of users on version A convert and 11% convert on version B, is that a real difference or random variation? Statistical significance testing tells you whether the observed difference is large enough (relative to the amount of data) to conclude version B is genuinely better. Without this test, you risk making changes based on lucky random variation rather than real improvements.
At home
The p-value and its meaning: the p-value is the probability of observing a result as extreme as yours if there were actually no real effect (the null hypothesis is true). A p-value of 0.03 means there is a 3% chance of seeing your result by random chance alone: below the 5% threshold, so the result is 'statistically significant'. Common misinterpretation: p=0.05 does NOT mean there is a 95% chance your hypothesis is true. It means: if the null hypothesis (no effect) were true, you would see results this extreme 5% of the time. These are different statements. The p-value says nothing about the probability that the null hypothesis is actually true.
At work
Statistical significance vs. practical significance: a result can be statistically significant but business-irrelevant. With large enough sample sizes (millions of users), tiny differences become statistically significant. A 0.01% conversion rate improvement that is statistically significant may not be worth implementing if the development cost exceeds the revenue impact. Practical significance means the effect is large enough to matter for business decisions: measured by effect size (how big is the difference?) rather than just p-value (could this be random?). Good A/B test analysis reports both: is this significant? and is this large enough to matter?
At home
The replication crisis and p-value problems: the p=0.05 threshold created incentives for 'p-hacking' (running analyses multiple ways until a significant result appears) and 'publication bias' (journals publishing significant results, discarding null results). These incentives mean the published scientific literature has more false-positive findings than the p=0.05 threshold implies. John Ioannidis's 2005 paper 'Why Most Published Research Findings Are False' formalized this concern mathematically. Proposed solutions include: pre-registration (specifying hypotheses and analysis methods before collecting data), higher significance thresholds (p<0.005 proposed by some researchers), Bayesian alternatives to p-values, and mandatory publication of null results.
At work
Sequential testing and A/B test peeking: a common error in business A/B testing is 'peeking': checking results before the planned end of the test and stopping when p<0.05 is reached. This inflates false positive rates because you are effectively running multiple tests. A test that reaches p=0.05 on day 3 of a planned 30-day test has a much higher than 5% false positive rate, because you would have stopped it at any of the 30 days it happened to be significant. Sequential testing methods (like SPRT or always-valid p-values) provide valid significance levels even when checking results continuously. Bayesian A/B testing frameworks are naturally resistant to the peeking problem.
Real-world examples
At home
Jerome read a news headline claiming that 'a new study shows eating chocolate significantly reduces risk of heart disease.' He knew 'statistically significant' meant the researchers found the result was unlikely to be random, but he also knew this did not mean the effect was large or that the study was definitive. He looked up whether the study was peer-reviewed, what its sample size was, and whether it had been replicated by other researchers. The actual effect size was tiny, the study was observational (not a clinical trial), and no other studies had confirmed it. The headline's use of 'significant' was technically accurate but misleading.
At work
Nadia managed A/B testing for an e-commerce platform. A junior analyst excitedly reported that a new checkout button colour had reached p=0.04 statistical significance after only 3 days. She recognized this as a peeking problem: they had planned a 14-day test. Stopping at day 3 when the result happened to be significant inflated the false positive rate. She continued the test to the planned end date. On day 14, p=0.23: the early significant result had been random variation. If they had implemented the change based on the day-3 result, they would have made a change based on a false positive.
Common confusions
- Many people assume 'statistically significant' means the result is important or large — Statistical significance only tells you the result is unlikely to be random: it says nothing about the size or practical importance of the effect. With large enough data, even trivially small effects become statistically significant. A drug that reduces systolic blood pressure by 0.5 mmHg in a study of 100,000 patients might reach p<0.001 (very significant) while providing essentially no clinical benefit. 'Significant' in statistics has a technical meaning (unlikely to be chance) that differs from its everyday meaning (important or large). Always check effect size alongside p-values.
- Some assume that a non-significant result (p>0.05) means there is no effect — A non-significant result means you did not find sufficient evidence of an effect, not that no effect exists. The result may be non-significant because the sample size was too small to detect a real but small effect (underpowered study), because the effect genuinely does not exist, or because the measurement was imprecise. 'Absence of evidence is not evidence of absence': a p=0.3 result does not tell you there is no effect; it tells you this study was not informative enough to detect one if it exists. Confidence intervals (showing the range of plausible effect sizes) communicate this ambiguity more clearly than p-values alone.
For analysts
When reviewing client A/B test results or research findings, statistical significance is a starting point, not an endpoint. The questions that follow: what was the effect size (is the difference large enough to matter commercially?), what was the sample size (was the test adequately powered to detect small effects?), was the test run for a predetermined duration without peeking?, and are the results consistent with other evidence? A/B testing frameworks (Optimizely, VWO, and in-house platforms; Google Optimize was shut down in 2023) vary in how they handle these issues: understanding their statistical methodology is important for interpreting results correctly.
History & origin
Ronald Fisher popularized the p-value in his 1925 book Statistical Methods for Research Workers as a convenient way to judge experimental evidence. The 0.05 threshold was Fisher's informal suggestion rather than a derived mathematical requirement. Jerzy Neyman and Egon Pearson developed a competing framework (hypothesis testing with explicit Type I and Type II error rates) that combined with Fisher's p-value to produce the modern statistical testing approach used in most research. The p=0.05 threshold became institutionalized in science, particularly in medicine and psychology, as a publication criterion: journals preferentially published statistically significant findings. This incentive structure contributed to the 'replication crisis' of the 2010s: many published significant findings (particularly in social psychology and nutrition) failed to replicate when other researchers ran the same experiments. In 2019, over 800 scientists signed a Nature commentary calling for 'retiring statistical significance' as a decision rule.