Multiple Comparisons: Bonferroni, Holm, FDR

Picture a study where you run a single t-test at the conventional threshold of alpha = 0.05. If the null hypothesis is true, you have a 5% chance of a false positive — a finding that looks real but is just noise. That risk is acceptable for one test. The trouble starts when you run many tests at once: comparing every pair of groups, screening dozens of genes, or correlating one outcome with twenty predictors. Each test carries its own 5% gamble, and those gambles accumulate fast.

This article explains why running many tests inflates false positives, the two different things you might want to control, and the three corrections you will actually use in practice: Bonferroni, Holm, and Benjamini-Hochberg (FDR). By the end you will know which one to reach for and how to report it.

Why multiple tests inflate false positives

Suppose you perform 20 independent tests and every null hypothesis is genuinely true. The probability that a single test does not produce a false positive is 0.95. The probability that none of the 20 misfires is 0.95^20 ≈ 0.36. So the probability of getting at least one false positive is 1 − 0.36 ≈ 0.64 — about a two-in-three chance of a spurious "significant" result. In expectation, 20 × 0.05 = 1 false positive will appear by chance alone.

Scale that to 100 tests and you expect five false positives, almost guaranteeing at least one. This is the multiple-comparisons problem: the more questions you ask of the same data, the more likely you are to be fooled by randomness. Correcting for it is not optional bureaucracy — it is the difference between a reproducible finding and a coincidence dressed up as a discovery.

Two error rates: FWER vs FDR

Before choosing a method, decide what you want to control. There are two distinct definitions of "too many false positives."

Family-wise error rate (FWER)

The FWER is the probability of making at least one false positive across the entire family of tests. Controlling FWER at 0.05 means: there is only a 5% chance that any of your rejections is wrong. This is strict, conservative, and appropriate when a single false claim is costly — confirmatory clinical endpoints, regulatory submissions, or a small set of pre-registered hypotheses.

False discovery rate (FDR)

The FDR is the expected proportion of false positives among the tests you declared significant. Controlling FDR at 0.05 means: on average, no more than 5% of your "discoveries" are false. You tolerate a few errors as long as the majority of your hits are real. This is the right frame for screening and exploratory work — gene expression, brain-imaging voxels, or scanning many candidate predictors — where the goal is to generate a shortlist for follow-up, not to bet everything on each individual result.

The key intuition: FWER controls a count (any error at all), FDR controls a rate (the fraction of errors among hits). FDR is more permissive, which is exactly why it keeps much more power when the number of tests is large.

Bonferroni: simple and conservative

The Bonferroni correction is the easiest to apply: divide your alpha by the number of tests m. With 20 tests, you compare each p-value to 0.05 / 20 = 0.0025 instead of 0.05. Equivalently, multiply each p-value by m to get an adjusted p-value (capped at 1) and compare it to the original 0.05.

Bonferroni controls the FWER, and it does so under any dependence structure between the tests. Its weakness is that it is needlessly strict: by forcing every test through the same tiny threshold it sacrifices power, especially when m is large or the tests are correlated. You will often fail to detect true effects that a better method would have caught. There is essentially never a reason to stop at plain Bonferroni when Holm is available.

Holm: same FWER, more power

Holm's step-down procedure controls the exact same FWER as Bonferroni but is uniformly more powerful — it rejects at least as many hypotheses, and usually more. The recipe:

1. Sort the m p-values from smallest to largest.
2. Compare the smallest to alpha / m, the next to alpha / (m − 1), the next to alpha / (m − 2), and so on.
3. Stop at the first p-value that fails the comparison; reject all hypotheses before it and retain that one and all the rest.

Because only the very smallest p-value faces the full Bonferroni threshold and later ones get progressively easier cutoffs, Holm recovers power Bonferroni throws away — for free, with no extra assumptions. Like Bonferroni, it holds under any dependence structure. The practical rule is simple: whenever you would use Bonferroni, use Holm instead.

Benjamini-Hochberg: controlling the FDR

When you have many tests and you are screening rather than confirming, Benjamini-Hochberg (BH) is the workhorse. It controls the FDR and retains far more power than any FWER method. The procedure:

1. Sort the m p-values ascending; let rank i run from 1 to m.
2. Find the largest i for which p(i) ≤ (i / m) × alpha.
3. Reject that hypothesis and all with smaller p-values.

The threshold grows with rank, so BH is dramatically less punishing than Bonferroni or Holm when many of your tests are truly non-null. The trade-off is that you accept a small expected fraction of false discoveries among your hits rather than near-certainty that none are false. One caveat: BH provably controls the FDR under independence or positive dependence between tests — the common case in screening. Under arbitrary dependence, use the more conservative Benjamini-Yekutieli variant.

Worked example

Suppose you compare a treatment against control on m = 5 outcome measures and obtain raw p-values: 0.001, 0.013, 0.021, 0.038, 0.045.

• Bonferroni (threshold 0.05/5 = 0.01): only 0.001 survives. One discovery.
• Holm: 0.001 vs 0.01 → reject; 0.013 vs 0.05/4 = 0.0125 → fails, stop. One discovery (here it matches Bonferroni, but on borderline data Holm often rejects more).
• BH (compare p(i) to i/5 × 0.05): 0.001≤0.01 ✓, 0.013≤0.02 ✓, 0.021≤0.03 ✓, 0.038≤0.04 ✓, 0.045≤0.05 ✓ — largest passing rank is 5, so all five are declared significant.

Same data, three answers: one, one, and five discoveries. The method is not a footnote — it changes your conclusions. You can run all three corrections automatically in MindStat and compare adjusted p-values side by side.

Which method should you use?

Reporting

Always state which correction you applied and report adjusted p-values (or the adjusted threshold), not just the raw ones. Write, for example, "p-values were adjusted for multiple comparisons using the Holm step-down method" or "FDR was controlled at 0.05 via Benjamini-Hochberg." This lets readers reproduce your decisions and judge how aggressive your screening was. While you are tidying your reporting, pair the corrected p-values with effect sizes — see How to Report Effect Size in APA Style.

Key takeaways

• Many tests inflate false positives: 20 tests at 0.05 expect about one spurious "significant" result, with a ~64% chance of at least one.
• FWER controls the probability of any false positive; FDR controls the expected proportion of false positives among rejections.
• Bonferroni (divide alpha by m) is simple and conservative; Holm controls the same FWER with strictly more power — prefer Holm.
• Benjamini-Hochberg controls FDR (under independence or positive dependence) and keeps far more power for many exploratory tests.
• Few confirmatory tests → Holm; many exploratory tests → BH. Report adjusted p-values and name the correction.

Choosing the right test in the first place matters just as much as correcting it afterward — see ANOVA vs Kruskal-Wallis: Which Test to Use and Checking Normality: Tests vs Plots Done Right.