ANOVA vs Kruskal-Wallis: Which Test to Use

When you have three or more independent groups and a numeric outcome, two tests dominate the decision: one-way ANOVA and the Kruskal-Wallis test. They answer related questions but rest on very different assumptions, and choosing the wrong one either wastes power or produces a misleading p-value. This guide explains exactly what each test does, what it assumes, and how to decide—then walks through a worked example end to end.

What each test actually compares

One-way ANOVA compares group means using an F-test. It partitions total variability into between-group and within-group components; if the between-group variance is large relative to the within-group variance, the F statistic is large and p is small. A significant result tells you that at least one group mean differs—not which one.

Kruskal-Wallis is the rank-based alternative. It replaces the raw values with their ranks across all groups combined, then tests whether the mean ranks differ. It is often described as a test of medians, but that interpretation is only clean when the group distributions share the same shape and spread. If shapes differ, a significant Kruskal-Wallis result means the distributions differ in some way (location, spread, or skew)—not necessarily that medians differ. This subtlety is widely misreported, so state your conclusion carefully.

The assumptions behind ANOVA

ANOVA's p-values are trustworthy when three conditions roughly hold:

• Independence of observations (design issue, not fixable after the fact).
• Approximately normal residuals—it is the residuals, not the raw outcome, that must be near-normal. Check with a Q-Q plot rather than relying on a single test. Beware the large-n trap: with big samples, a Shapiro-Wilk test flags trivial, harmless departures as "significant," while with small samples it has little power to detect real ones—so let the plot lead. See our guide on Checking Normality: Tests vs Plots Done Right.
• Homogeneity of variance across groups, assessed with Levene's test (more robust than Bartlett's when data are non-normal).

ANOVA is fairly robust to mild non-normality, especially with balanced designs and moderate sample sizes, thanks to the Central Limit Theorem acting on group means. It is far less forgiving of unequal variances combined with unequal group sizes, where the Type I error rate can drift well away from your nominal level.

When variances are unequal: Welch's ANOVA

If Levene's test flags heterogeneity, do not abandon means—switch to Welch's ANOVA, which uses a Welch–Satterthwaite adjustment to the degrees of freedom and does not assume equal variances. Many methodologists recommend Welch as a sensible default for one-way designs, mirroring the logic behind preferring Welch's over Student's t-Test in the two-group case.

When to reach for Kruskal-Wallis

Kruskal-Wallis is the better choice when:

• The outcome is ordinal (e.g., Likert ratings) where means are not meaningful.
• Distributions are strongly skewed or heavy-tailed and cannot be reasonably transformed.
• There are influential outliers that distort means—ranks dampen their effect.
• Sample sizes are small and normality cannot be assessed or assumed.

Because it uses ranks, Kruskal-Wallis sacrifices a little power when ANOVA's assumptions genuinely hold (its asymptotic relative efficiency is about 0.95 under normality), but it is remarkably robust to non-normality and outliers. One caveat: Kruskal-Wallis is not a cure for unequal variances—like ANOVA, it can be distorted when groups have very different spreads, so heterogeneity alone is not a reason to switch to it.

Quick comparison

Effect sizes—always report one

A p-value tells you whether an effect exists, not how large it is. For ANOVA, report eta-squared (η²)—the proportion of variance explained—or the less biased omega-squared (ω²), which is preferable in small samples. For Kruskal-Wallis, report epsilon-squared (ε²) or eta-squared based on the H statistic (η²_H). Conventional benchmarks (small ≈ 0.01, medium ≈ 0.06, large ≈ 0.14) are starting points, not rules.

Post-hoc comparisons

A significant omnibus test only says groups differ somewhere. To locate the differences, follow ANOVA with Tukey's HSD, which controls the family-wise error rate across all pairwise comparisons. After Kruskal-Wallis, use Dunn's test with a correction such as Holm or Benjamini-Hochberg. Holm controls the same family-wise error rate as Bonferroni but is uniformly more powerful, while Benjamini-Hochberg controls the false discovery rate—a less strict criterion that yields more power when you have many comparisons. Either way, you must adjust for multiplicity—see Multiple Comparisons: Bonferroni, Holm, FDR.

A worked decision example

Suppose you compare recovery time (days) across three rehabilitation protocols, n = 12 per group. You plot the residuals: Group C is right-skewed with two long-recovery outliers, and Levene's test returns p = 0.03, signalling unequal variances. Two assumptions are shaken at once.

The decision tree: residuals are clearly non-normal and influenced by outliers, so plain ANOVA on means is risky. You have two defensible paths. If you still care about means and the skew is moderate, Welch's ANOVA handles the unequal variances; combined with a robust check, it may suffice. But given the outliers and skew, the cleaner choice is Kruskal-Wallis. Running it yields H(2) = 9.84, p = 0.007, ε² = 0.28 (a large effect). Because the three distributions have visibly different shapes, you report this as "the protocols differ in recovery-time distribution," not strictly "median recovery differs." Dunn's test with Holm correction then shows Protocol C differs significantly from A and B, while A and B do not differ.

You can run all of this—Levene's test, Welch's ANOVA, Kruskal-Wallis, and Dunn's post-hoc with correction—directly in MindStat and compare the outputs side by side before committing to one analysis.

Key takeaways

• Check assumptions first: Q-Q plot of residuals plus Levene's test, not a reflex choice.
• Approximately normal, equal variance → one-way ANOVA on means.
• Normal but unequal variance → Welch's ANOVA.
• Skewed, ordinal, outliers, or tiny n → Kruskal-Wallis.
• Interpret Kruskal-Wallis carefully: "medians differ" only if shapes match; otherwise "distributions differ."
• Always report an effect size and follow a significant omnibus test with corrected post-hoc comparisons (Tukey HSD or Dunn's).