Checking Normality: Tests vs Plots Done Right

Few habits cause more confusion among graduate students than the ritual of "checking normality." Researchers run a Shapiro-Wilk test, see p < .05, panic, and abandon a perfectly good t-test for a rank-based alternative they barely understand. Others stare at a slightly lopsided histogram and declare the assumption violated. The truth is more nuanced: normality matters less than most people think, it applies to a different quantity than most people check, and a single significance test is the worst tool for the job.

It's the residuals, not the raw variable

The first and most common mistake is testing the wrong thing. For a t-test, ANOVA, or linear regression, the normality assumption is about the residuals (the errors), not the raw outcome variable. Your dependent variable can be visibly skewed across the whole sample yet still produce beautifully normal residuals once group means or predicted values are accounted for.

Consider an experiment with two groups whose means differ substantially. The pooled outcome looks bimodal and clearly non-normal. But that "non-normality" is just the difference between groups — exactly the signal you are testing for. What the model assumes is that the deviations within each group (the residuals) are approximately normal. So in regression and ANOVA, fit the model first, then examine the residuals. For a simple two-group comparison, examine each group separately rather than the combined distribution.

Shapiro-Wilk: objective, but treacherous at the extremes

Shapiro-Wilk is among the most powerful general tests of normality and gives you an objective, reproducible p-value. That objectivity is seductive, but the test has two failure modes that trap researchers at opposite ends of the sample-size scale.

The large-n trap

Hypothesis tests detect any departure from the null, and with enough data they detect departures so tiny they are statistically real but practically irrelevant. At n = 5000, Shapiro-Wilk will flag a distribution that is, for all modeling purposes, indistinguishable from normal. A significant result here tells you almost nothing useful — it is a measure of your sample size as much as of your data. Reporting "normality was violated (p < .001)" on a large sample is technically true and analytically meaningless.

The small-n trap

At the other extreme, with n = 15 or n = 20, the test is badly under-powered. It will happily return p = .40 for data drawn from a genuinely skewed or heavy-tailed distribution simply because there is not enough information to reject. So a non-significant Shapiro-Wilk on a small sample is not evidence of normality — it is mostly evidence that you do not have much data. This is the exact situation where the assumption matters most and the test helps least.

Plots usually tell you more

A p-value collapses the entire question of distributional shape into one number. Plots restore the detail you actually need to make a judgment.

The Q-Q plot

The quantile-quantile (Q-Q) plot is the single most informative diagnostic. It plots your sorted (residual) values against the quantiles you would expect from a normal distribution. If the data are normal, the points fall along a straight diagonal line. More importantly, the pattern of departure is interpretable: an S-shape signals heavy or light tails, a curve that bends away at one end signals skew, and isolated points drifting off the line at the extremes flag outliers. A Q-Q plot tells you not just whether the data deviate but where and how — information no test statistic provides.

The histogram

Histograms are intuitive and good for communicating to a general audience, but they are bin-dependent: the same data can look bell-shaped with one bin width and ragged with another. Use a histogram to build intuition, but lean on the Q-Q plot for the actual judgment.

The Central Limit Theorem is your safety net

Here is the reassurance most checklists omit: inference about means is remarkably robust to non-normality once the sample is reasonably large. By the Central Limit Theorem, the sampling distribution of the mean approaches normality as n grows, even when the underlying data are skewed. For roughly n ≥ 30 per group (more if the skew is severe), t-tests and ANOVA hold their nominal error rates well despite non-normal residuals. The normality assumption is therefore most critical precisely when you have a small sample — the same regime where Shapiro-Wilk cannot reliably detect a problem. This is why blind reliance on the test is so misguided.

What to do when normality genuinely fails

If your plots and judgment agree that the departure is real and material — heavy skew, severe outliers, a small sample — you have three solid options:

• Use a more robust procedure. For comparing two groups, Welch's t-test relaxes the equal-variance assumption (a separate concern from normality, but often co-occurring) and is a sensible default; see Welch's vs Student's t-Test: When Variances Differ for when this matters.
• Transform the variable. A log or square-root transform can pull a right-skewed outcome back toward symmetry — but it changes the scale of interpretation, so report results on the transformed scale honestly.
• Switch to a rank-based test. The Mann-Whitney U test (two groups) and the Kruskal-Wallis test (three or more) make no normality assumption. See ANOVA vs Kruskal-Wallis: Which Test to Use for the trade-offs.

A worked example

Suppose you compare reaction times (ms) between a control and a treatment group, n = 18 each. Shapiro-Wilk on the residuals returns p = .03. Do you abandon the t-test? Not yet. You plot the residuals: the Q-Q plot shows a clean diagonal except for two points pulling away at the upper tail — classic right skew from a couple of slow responses, which is expected for reaction-time data. The histogram confirms a long right tail.

Here the small sample means the assumption genuinely matters, and the plot confirms a real, interpretable departure. The sensible move is to log-transform the reaction times (standard practice for this measure) and re-fit, or report a Mann-Whitney U test as a robustness check. You would not simply report "the data were non-normal" and stop. When you report the final comparison, pair it with an effect size — see How to Report Effect Size in APA Style.

A decision table

Key takeaways

• Check the residuals, not the raw outcome — for regression and ANOVA, fit first, then diagnose.
• Shapiro-Wilk is hyper-sensitive at large n and under-powered at small n; never let it decide alone.
• A Q-Q plot is usually more informative than any test; histograms are intuitive but bin-dependent.
• Thanks to the Central Limit Theorem, inference about means is robust at larger samples; normality matters most when n is small.
• If normality genuinely fails, reach for Welch's test, a transformation, or a rank-based alternative — and combine visual evidence, the test, and sample-size judgment rather than trusting one p-value.

You can fit a model, generate a Q-Q plot, and run Shapiro-Wilk side by side on the same data in MindStat, which makes it easy to weigh the visual against the test instead of obeying a single threshold.