The Multiple Comparison Problem
Interactive ANOVA Calculator
Compare three groups with ANOVA! Enter comma-separated values for each group and click Calculate F. The visualizer shows group means as bars and computes the F-statistic, degrees of freedom, p-value, and eta-squared effect size. Try making groups similar or very different to see how F changes.
The Problem with Multiple t-Tests
With 4 groups, you'd need 6 t-tests. Each has a 5% false positive rate. The chance of at least one false positive becomes 1 - (0.95)⁶ = 26%! This is the 'multiple comparison problem.'
Family-Wise Error Rate
| Groups | Comparisons | Family-Wise Error |
|---|---|---|
| 3 | 3 | 14% |
| 4 | 6 | 26% |
| 5 | 10 | 40% |
| 10 | 45 | 90% |
Family-wise error rate (FWER) is the probability of at least one false positive across all comparisons. With 10 groups, you're almost guaranteed a false positive using multiple t-tests!
ANOVA: One Omnibus Test
ANOVA asks one question: 'Is there ANY difference among the group means?' If yes, then you do follow-up tests. This keeps Type I error controlled at α for the initial test.
The F-Statistic
Between vs Within Variance
F compares two types of variance: (1) Between-groups: how much do group means differ from the grand mean? (2) Within-groups: how much do individuals vary within their groups?
What Makes F Large?
Large F means groups are well-separated relative to individual variation. Small F means groups overlap heavily. Think of it as signal-to-noise ratio.
Reading the ANOVA Table
| Source | SS | df | MS | F | p |
|---|---|---|---|---|---|
| Between | 120 | k-1 = 2 | 60 | 5.0 | 0.012 |
| Within | 324 | N-k = 27 | 12 | - | - |
| Total | 444 | N-1 = 29 | - | - | - |
The ANOVA table partitions total variance into between-group and within-group components. F = MS_between / MS_within. If p < α, at least one group mean differs significantly.
ANOVA Hypotheses
H₀: All group means are equal. Hₐ: At least one mean is different (not necessarily ALL different). Rejecting H₀ doesn't tell you WHICH groups differ—that requires post-hoc tests.
Post-Hoc Tests
After a Significant F
A significant ANOVA tells you 'something differs' but not what. Post-hoc tests make all pairwise comparisons while controlling family-wise error rate.
Popular Post-Hoc Tests
| Test | Control | Power | Use When |
|---|---|---|---|
| Tukey HSD | Strict | Moderate | Equal n, all pairs |
| Bonferroni | Very strict | Lower | Few comparisons |
| Scheffé | Very strict | Lowest | Complex contrasts |
| Games-Howell | Moderate | Good | Unequal variances |
Tukey's HSD is most common for all pairwise comparisons. Bonferroni is simpler but conservative. Scheffé allows complex contrasts. Games-Howell doesn't assume equal variances.
The Bonferroni Correction
Bonferroni divides α by the number of comparisons. Simple but conservative—with many groups, the adjusted α becomes so small that real differences may be missed.
How to Report Results
"A one-way ANOVA revealed a significant effect of treatment on test scores, F(2, 57) = 8.42, p < .001, η² = .23. Post-hoc comparisons using Tukey's HSD indicated that Group A (M = 85.2, SD = 7.1) scored significantly higher than Group B (M = 76.4, SD = 8.3), p = .003, and Group C (M = 74.1, SD = 6.9), p < .001. Groups B and C did not differ significantly (p = .58)."
Report: F(df_between, df_within) = value, p-value, effect size. Then describe post-hoc results specifying which pairs differed and which didn't.
Effect Size: Eta-Squared
η² (Eta-Squared)
Eta-squared is the proportion of total variance explained by group membership. It's analogous to R² in regression. η² = 0.35 means 35% of score differences are due to group differences.
How Big is Big?
Cohen's guidelines: η² = .01 (small), .06 (medium), .14 (large). But context matters—in some fields, η² = .04 might be practically important.
Omega-Squared: Less Biased
• Overestimates population effect
• Sample-specific
• Better for small samples
• Preferred by some journals
η² tends to overestimate the true population effect, especially with small samples. ω² (omega-squared) provides a less biased estimate. Some journals prefer ω².
Real-World Examples
Apply your knowledge to practical scenarios
Comparing Teaching Methods
A school tests three teaching methods: Traditional lecture (Group A, n=25, x̄=72), Interactive (Group B, n=25, x̄=81), and Online (Group C, n=25, x̄=78). Total N=75. ANOVA gives F(2,72) = 8.45, p = 0.001, η² = 0.19.
Sample Data
{
"groupA": {
"n": 25,
"mean": 72
},
"groupB": {
"n": 25,
"mean": 81
},
"groupC": {
"n": 25,
"mean": 78
},
"F": 8.45,
"df_between": 2,
"df_within": 72,
"p": 0.001,
"eta_squared": 0.19
}Key Insights
- ANOVA answers 'is there ANY difference?' not 'which groups differ?'
- Post-hoc tests (Tukey HSD) are needed to identify specific differences
- Effect size (η²) helps assess practical importance beyond p-value
The Danger of Multiple t-Tests
A researcher wants to compare 5 training programs. Instead of ANOVA, they run all possible t-tests: 10 comparisons. They find 2 'significant' results at α = 0.05.
Key Insights
- 10 t-tests at α = 0.05 gives ~40% false positive rate overall
- ANOVA keeps family-wise error at α for the omnibus test
- Post-hoc corrections (Tukey, Bonferroni) are essential for pairwise comparisons
Drug Dosage Optimization
A pharmaceutical study tests 4 doses of a pain reliever: Placebo (0mg), Low (100mg), Medium (200mg), High (400mg). N = 20 per group, 80 total. Pain reduction scores recorded. ANOVA: F(3,76) = 12.3, p < 0.001.
Key Insights
- Significant ANOVA doesn't mean all groups differ—pattern could be complex
- Post-hoc tests reveal the actual structure of group differences
- Ceiling effects and practical considerations matter beyond statistics
ANOVA Quiz
Test your understanding • Passing score: 70%