What is Effect Size?
Interactive Effect Size Calculator
Calculate Cohen's d effect size! Enter the means for two groups and the pooled standard deviation to see how large the effect is. The visualization shows two overlapping distributions—the more they separate, the larger the effect. Cohen's guidelines: 0.2 = small, 0.5 = medium, 0.8 = large.
Beyond P-Values
P-values tell you IF an effect exists, but not HOW BIG it is. With large samples, even tiny, meaningless differences become 'significant.' Effect sizes quantify the magnitude of effects, helping you judge practical importance.
Significance vs. Importance
| Group | Recovery Rate | Difference |
|---|---|---|
| Drug | 52.1% | p < 0.001 d = 0.04 (tiny) |
| Placebo | 50.0% |
With 100,000 participants, a 2.1% difference is highly significant statistically. But is it clinically meaningful? Effect size helps answer this. A drug with side effects might need a larger benefit to be worthwhile.
Common Effect Size Measures
Different effect size measures suit different analyses. Cohen's d compares group means, correlation (r) measures relationships, odds ratios work for categorical outcomes, and eta-squared measures variance explained in ANOVA.
Cohen's d
Calculating Cohen's d
Cohen's d expresses the difference between two means in standard deviation units. A d of 0.50 means the groups differ by half a standard deviation. This standardization allows comparison across different studies and measures.
Cohen's Conventions
| Size | d Value | % Overlap | Interpretation |
|---|---|---|---|
| Small | 0.2 | 85% | Subtle, hard to notice |
| Medium | 0.5 | 67% | Noticeable difference |
| Large | 0.8 | 53% | Obvious difference |
Jacob Cohen proposed these conventions: d=0.2 is small (hard to see with naked eye), d=0.5 is medium (visible), d=0.8 is large (obvious). But context matters - a 'small' effect might be important for life-saving interventions!
Overlapping Distributions
With d=0.5, about 67% of the two distributions overlap. This means there's substantial overlap between groups - a randomly selected person from the higher-scoring group only beats someone from the lower group 64% of the time.
Statistical Power
Interactive Power Calculator
Explore how power depends on effect size, sample size, and alpha! Adjust the sliders to see the power curve change. The red dot shows your current power level. Notice how power increases rapidly at first with larger n, then plateaus. The green dashed line marks the 80% power threshold.
What is Power?
| Effect Exists (H₁ true) | No Effect (H₀ true) | |
|---|---|---|
| Reject H₀ (p < .05) | Correct! (Power = 1-β) | Type I Error (α = 0.05) |
| Fail to reject H₀ (p ≥ .05) | Type II Error (β) | Correct! (1-α) |
Power is the probability of correctly rejecting H₀ when it's false - i.e., detecting a real effect. Low power means you might miss real effects (Type II error). The convention is to aim for power ≥ 0.80 (80%).
What Affects Power?
Four factors determine power: sample size, effect size, significance level (α), and variability. Since effect size and variability are often fixed, researchers typically adjust sample size to achieve desired power.
The Problem of Low Power
Underpowered studies are problematic: they usually miss real effects, and when they do find significance, effects are often exaggerated. This contributes to the replication crisis in science.
Power Analysis Example
| Test Type | N per group | Total N |
|---|---|---|
| Independent t-test | 64 | 128 |
| Paired t-test | 34 | 34 |
| One-way ANOVA (3 groups) | 52 | 156 |
| Correlation | - | 84 |
Power analysis calculates the sample size needed to detect an effect of a given size with specified power. For a medium effect (d=0.5), you typically need 64 participants per group for an independent t-test.
Sample Size Determination
Interactive Sample Size Calculator
Calculate the sample size you need! Set your expected effect size, desired power, and significance level to see the required n per group. Notice how smaller effect sizes require much larger samples. This is why pilot studies and effect size estimates are crucial for planning research.
Planning Sample Size
A priori power analysis determines sample size BEFORE data collection. You specify: (1) expected effect size (from pilot or literature), (2) desired power (usually 80%), (3) alpha level (usually 0.05), and (4) test type.
Minimum Detectable Effect
Sensitivity analysis asks: given my fixed sample size, what's the smallest effect I can detect with adequate power? Useful when sample size is constrained by budget or available participants.
Achieved Power
Post-hoc power calculates achieved power after a study. But it's controversial because it's mathematically tied to the p-value - a non-significant result will always show low post-hoc power. Confidence intervals are more informative.
Other Effect Size Measures
r as Effect Size
| Size | r Value | r² (Variance) | Example |
|---|---|---|---|
| Small | 0.10 | 1% | Height & IQ |
| Medium | 0.30 | 9% | Income & education |
| Large | 0.50 | 25% | Height & weight |
Correlation (r) is itself an effect size measure. Cohen's conventions: r=0.10 small, r=0.30 medium, r=0.50 large. Note that r² (variance explained) is small even for 'large' correlations - r=0.50 means only 25% shared variance!
η² and Partial η²
| Size | η² | Partial η² |
|---|---|---|
| Small | 0.01 | 0.01 |
| Medium | 0.06 | 0.06 |
| Large | 0.14 | 0.14 |
For ANOVA, eta-squared (η²) measures variance explained by the effect. Partial η² divides by effect + error variance, ignoring other factors. Partial η² is larger and more commonly reported, but can be misleading in complex designs.
Effect Sizes for Categorical Data
| Recovered | Not Recovered | |
|---|---|---|
| Treatment | 80 | 20 |
| Control | 50 | 50 |
The odds ratio compares odds of an outcome between groups. OR=1 means no effect, OR>1 means higher odds in the first group. For rare outcomes, OR approximates relative risk. OR=4.0 is generally considered a large effect.
Real-World Examples
Apply your knowledge to practical scenarios
Antidepressant Clinical Trial
A pharmaceutical company tests a new antidepressant. With 5,000 participants, they find p < 0.001. However, the drug reduces depression scores by only 2 points on a 0-100 scale (d = 0.15). The control group improves by 8 points, the drug group by 10 points.
Sample Data
{
"control_improvement": 8,
"drug_improvement": 10,
"difference": 2,
"p_value": "< 0.001",
"cohens_d": 0.15,
"sample_size": 5000
}Key Insights
- Statistical significance doesn't equal clinical significance
- Always report effect sizes alongside p-values
- Large samples can make trivial effects 'significant'
Educational Intervention Planning
A school district wants to evaluate a new math tutoring program. Previous research suggests tutoring programs typically produce effects of d = 0.40. They want 80% power to detect this effect at α = 0.05 using a two-sample t-test.
Sample Data
{
"expected_effect": "d = 0.40",
"desired_power": 0.8,
"alpha": 0.05,
"test": "Independent t-test",
"calculated_n_per_group": 100,
"total_n": 200
}Key Insights
- Power analysis should be done before data collection
- Smaller expected effects require larger samples
- Underpowered studies risk missing real effects
Website A/B Test Power
An e-commerce company wants to test a new checkout design. Current conversion rate is 3%. They want to detect a 0.5 percentage point increase (to 3.5%) - a 17% relative improvement. They get 100,000 visitors per week.
Sample Data
{
"baseline_conversion": "3.0%",
"target_conversion": "3.5%",
"relative_lift": "17%",
"weekly_traffic": 100000,
"required_sample_per_variant": 15000,
"test_duration": "~2 days"
}Key Insights
- Business impact depends on both effect size and scale
- Pre-register sample size to avoid peeking problems
- Small effects can have large business value at scale
Effect Sizes & Power Quiz
Test your understanding • Passing score: 70%