Setting Up Hypotheses
The Null Hypothesis
• The new drug has no effect
• The mean equals 100
• Groups are the same
H₀ (null hypothesis) claims there's no effect, no difference, or no relationship. It represents the skeptical position—assuming nothing interesting is happening until proven otherwise.
The Alternative Hypothesis
• The new drug is effective
• The mean ≠ 100
• Groups are different
Hₐ (alternative hypothesis) is what you're trying to prove—that there IS an effect, difference, or relationship. This is usually the research hypothesis you care about.
One-Tailed vs Two-Tailed
Effect in either direction
Effect in one direction only
Two-tailed tests check for effects in EITHER direction (higher or lower). One-tailed tests only check ONE direction. Use two-tailed unless you have strong theoretical reason for one-tailed.
The Testing Logic
We can't prove H₀ or Hₐ directly. Instead, we assume H₀ is true, then ask: 'How unlikely is our observed data?' If very unlikely under H₀, we reject H₀ in favor of Hₐ.
The p-value
Interactive p-Value Simulator
Run your own hypothesis test simulation! Set a sample size and hypothesized mean (H₀), then click 'Run Simulation' to generate random data and calculate the t-statistic and p-value. Run it multiple times to see how p-values vary across samples. Notice how larger sample sizes tend to produce more consistent results.
What is a p-value?
The p-value is the probability of observing data as extreme (or more extreme) than what we got, IF H₀ is true. A small p-value means our data is unlikely under H₀.
Reading p-values
Smaller p-values = stronger evidence against H₀. But p-values are continuous—don't treat 0.049 and 0.051 as fundamentally different. Consider the actual value, not just whether it crosses a threshold.
What p-value is NOT
p is about data, not hypotheses
Results are what they are; p measures surprise
Small p ≠ large effect (could be large n)
p-value is NOT the probability H₀ is true. It's the probability of our data IF H₀ were true. Also, p doesn't measure effect size—a tiny effect can have p < 0.001 with large enough n.
Where p Comes From
We convert our data to a test statistic, then find how much of the theoretical distribution lies beyond that value. This tail area is the p-value—probability of being that extreme or more.
Significance Level (α)
The Decision Threshold
α (alpha) is the threshold you set BEFORE the test. If p < α, you reject H₀. If p ≥ α, you fail to reject H₀. Common values: 0.05 (5%), 0.01 (1%), 0.10 (10%).
Choosing Alpha
| α | When to Use | Field |
|---|---|---|
| 0.10 | Exploratory research | Social science |
| 0.05 | Standard threshold | Most fields |
| 0.01 | Higher stakes | Medicine |
| 0.005 | New proposed standard | Psychology |
α = 0.05 is conventional but arbitrary. Some argue it's too lenient (contributing to replication crisis). Consider the stakes: medical decisions might warrant α = 0.01 or lower.
Statistical Significance
Evidence supports Hₐ
Insufficient evidence
If p < α, we say the result is 'statistically significant'—unlikely to occur by chance if H₀ is true. But 'significant' doesn't mean 'important' or 'large effect'—just unlikely under H₀.
Proper Conclusions
"We fail to reject H₀"
"The data provide sufficient evidence to..."
"We prove Hₐ" (can't prove anything)
"H₀ is true" (we only assess evidence)
Never say 'accept H₀' or 'prove' anything. We can only reject H₀ or fail to reject it. Failing to reject doesn't mean H₀ is true—we just don't have enough evidence against it.
Type I & Type II Errors
Two Types of Mistakes
| H₀ True | H₀ False | |
| Reject H₀ | Type I Error False Positive P = α | Correct! True Positive P = 1-β (Power) |
| Fail to Reject | Correct! True Negative P = 1-α | Type II Error False Negative P = β |
Type I: Rejecting H₀ when it's actually true (false alarm). Type II: Failing to reject H₀ when it's actually false (missed detection). Both are mistakes with real consequences.
False Positive (Type I)
Conclusion: "Drug works!" (Wrong!)
Consequence: Patients get ineffective treatment
Probability: α (e.g., 5%)
Type I error is a 'false positive'—claiming an effect exists when it doesn't. Setting α = 0.05 means you accept a 5% chance of making this error when H₀ is true.
False Negative (Type II)
Conclusion: "No effect found" (Wrong!)
Consequence: Effective drug not approved
Probability: β (often 10-20%)
Type II error is a 'false negative'—failing to detect a real effect. This happens when sample size is too small or effect is subtle. β = P(Type II), and Power = 1 - β.
The Error Tradeoff
Type I and Type II errors trade off: lowering α (stricter) reduces false positives but increases false negatives. The only way to reduce BOTH is to increase sample size.
Real-World Examples
Apply your knowledge to practical scenarios
The New Drug Dilemma
A pharmaceutical company tests a new heart medication. They set α = 0.05 and conduct the trial. The result: p = 0.048. The FDA must decide whether to approve the drug.
Key Insights
- p < 0.05 is not 'proof'—it's evidence against H₀ assuming a 5% false positive tolerance
- Multiple testing inflates false positive rate dramatically
- FDA requires replication and clinical significance, not just statistical significance
Type I and Type II Errors in Court
The criminal justice system parallels hypothesis testing. H₀: Defendant is innocent. Hₐ: Defendant is guilty. 'Beyond reasonable doubt' is like setting α very low.
Key Insights
- Type I and Type II errors have real-world consequences
- Society implicitly sets α based on which error is more costly
- The only way to reduce BOTH errors is better evidence (larger sample/better data)
COVID Test Interpretation
A COVID rapid test has 95% sensitivity (true positive rate) and 99% specificity (true negative rate). In a population where 1% actually have COVID, you test positive. What's the probability you actually have COVID?
Key Insights
- P(A|B) ≠ P(B|A) — the direction of conditioning matters enormously
- Base rates (prevalence) dramatically affect interpretation
- p-value tells you about the data assuming H₀, NOT about H₀ given the data
Hypothesis Testing Quiz
Test your understanding • Passing score: 70%