Hypothesis Testing
Also known as: null hypothesis, alternative hypothesis, A/B testing statistics, p-value testing, Type I error, Type II error
Hypothesis testing is a formal statistical procedure for evaluating whether observed data provides sufficient evidence against a baseline assumption (the null hypothesis).
The process: define a null hypothesis (H0, typically 'no effect' or 'no difference'), define an alternative hypothesis (H1, the effect or difference you believe might exist), collect data, calculate a test statistic, and determine the probability (p-value) of observing data this extreme if the null hypothesis were true. If the p-value falls below a predetermined threshold (alpha, typically 0.05), the null hypothesis is rejected in favor of the alternative. Hypothesis testing is the statistical framework underlying A/B tests, clinical trials, quality control checks, and most scientific experiments.
Hypothesis Testing, explained at three levels
At home
Hypothesis testing is how scientists and researchers decide whether an experiment showed a real result or just random luck. The process starts with a hypothesis (a specific guess about what will happen) and a null hypothesis (the boring alternative: 'nothing is happening'). After running the experiment, they ask: if nothing were actually happening, how likely would it be to see results as extreme as ours? If that probability is very low (less than 5%), they conclude something probably IS happening. It is like a legal standard: you start assuming 'not guilty' (null hypothesis) and only convict (reject null hypothesis) if the evidence is strong enough.
At work
A/B testing is hypothesis testing applied to business decisions. The null hypothesis is 'version A and version B perform the same' (no real difference). You run the test and collect data. If the conversion rate difference is large enough and statistically significant (p<0.05), you reject the null hypothesis and conclude version B is genuinely better. Without this formal testing, you might mistake random variation for a real improvement -- implementing a change that appears better by chance and then wondering why it doesn't continue to outperform.
At home
Type I and Type II errors: Type I error (false positive) is rejecting the null hypothesis when it is actually true -- concluding there is an effect when there isn't one. With alpha=0.05, you expect 5% of tests with no real effect to produce false positives. Type II error (false negative) is failing to reject the null hypothesis when it is actually false -- missing a real effect. Statistical power is the probability of correctly detecting a real effect (avoiding Type II error). A test with 80% power will miss 20% of real effects. There is always a trade-off: making it harder to get a false positive (lower alpha) makes it more likely you miss real effects (lower power).
At work
One-tailed vs. two-tailed tests: a two-tailed test asks 'is there any difference in either direction?' -- appropriate when you genuinely don't know if the effect could go either way. A one-tailed test asks 'is the effect in this specific direction?' -- appropriate when you have strong prior reason to expect the effect can only go one way. One-tailed tests have more statistical power (easier to reach significance) but are only valid when the direction is predetermined before collecting data. Using a one-tailed test because the data came out in the expected direction (post-hoc) is a form of p-hacking and inflates false positive rates.
At home
Multiple comparisons problem: when you run many hypothesis tests simultaneously (testing 20 different variants, checking 50 website metrics at once, analyzing data across many demographic subgroups), the probability of at least one false positive increases dramatically. With 20 independent tests at alpha=0.05, you expect 1 false positive even if there are no real effects. The Bonferroni correction (dividing alpha by the number of tests) is a simple but conservative correction. False Discovery Rate (FDR) control methods (like Benjamini-Hochberg) are more powerful alternatives. This problem is pervasive in business analytics when teams test many metrics simultaneously.
At work
Pre-registration as a countermeasure to hypothesis testing inflation: pre-registration commits researchers to their hypothesis, analysis plan, and stopping rule before collecting data -- making post-hoc p-hacking visible. Journal pre-registered reports accept papers before results are known, ensuring null results are published (eliminating publication bias). In business A/B testing, the equivalent is specifying your primary metric, sample size plan, and analysis approach in a test plan document before launching the test -- and not adding metrics or changing the analysis after seeing early results. Pre-registration reduces false positives but requires accepting that some experiments will be underpowered or that interesting patterns cannot be retroactively analyzed as primary findings.
Real-world examples
At home
Marcus read that a study found people who drank coffee lived longer. The researchers had tested 80 different dietary habits and found coffee consumption was the only one reaching p<0.05 for longevity. He recognized this as a multiple comparisons problem: with 80 tests at p=0.05, you expect about 4 false positives even if none of the dietary habits actually affected longevity. The coffee finding might be one of those 4 false positives. A single study testing one pre-specified hypothesis about coffee and longevity would be much more credible evidence than a mining operation testing 80 habits.
At work
Aisha ran a product analytics team. After a major feature release, an analyst pulled 45 engagement metrics and reported that 3 were statistically significantly improved at p<0.05. She recognized the multiple comparisons problem: with 45 metrics at alpha=0.05, you expect about 2.25 false positives by chance alone. Before the test, they had identified 3 primary metrics that the feature was designed to improve. Those 3 metrics all showed improvement (which was genuine evidence of success). The 3 other 'significant' metrics were in the exploratory category and would require further confirmation. She reported results clearly distinguishing pre-specified primary metrics from exploratory findings.
Common confusions
- Many people assume that failing to reject the null hypothesis (p>0.05) proves there is no effect — Failing to reject the null hypothesis means you do not have sufficient evidence of an effect -- it does not mean the effect does not exist. The study may have been underpowered (too small to detect a real but small effect), the effect may be in a direction not tested, or the measurement may have been too imprecise. 'We found no significant evidence of harm' is not the same as 'we found evidence of no harm'. This distinction matters enormously in regulatory contexts (showing a drug is safe requires evidence of absence, not just absence of evidence) and in business decisions where null results from small tests are often mistakenly treated as evidence that a strategy doesn't work.
- Some assume that a single significant A/B test result is sufficient to confidently implement a change — A single test with p<0.05 has a 5% false positive rate by construction -- meaning if you run 20 A/B tests, you expect 1 to show a false positive 'win'. Implementing all 'significant' single-test results will include some false positives. Best practice includes: running the test for a pre-determined duration (not stopping early when significant), having a pre-specified primary metric, considering whether the effect size is practically meaningful, and for high-stakes decisions, replicating the test or seeking corroboration from other evidence before fully committing. Single-test significance is better understood as 'promising enough to warrant further investigation' than 'definitively proven'.
For analysts
When clients report A/B test or research results, a quick hypothesis testing audit adds value: was the primary metric specified before the test started (or chosen after seeing results)? Was the test run for the pre-planned duration (or stopped early when significance appeared)? How many metrics were tested (is multiple comparisons adjustment needed)? Were the sample sizes adequate (was the test powered to detect the effect sizes that would matter for the business decision)? These questions distinguish rigorous testing from fishing for favorable results -- and help clients build the evidence quality needed for confident business decisions.
History & origin
The modern hypothesis testing framework synthesizes Ronald Fisher's p-value approach (1920s) and the Neyman-Pearson framework (1930s) of explicitly defined alternative hypotheses, Type I and Type II errors, and decision rules. Fisher viewed the p-value as a continuous measure of evidence; Neyman and Pearson advocated binary accept/reject decisions with pre-specified error rates. The synthesis used today combines both, creating a framework that neither Fisher nor Neyman-Pearson originally endorsed. This hybrid is the dominant approach in applied science and business. Jacob Cohen's work on statistical power (1962, 1988) established that most research was underpowered (too small to detect plausible effects), a problem that contributed to the replication crisis. The American Statistical Association issued a statement in 2016 cautioning against binary decision-making based on p=0.05 alone.