The Multiple Regression Equation
Adding More Predictors
Salary = 30,000 + 2,500(Years) + 5,000(Degree) + 1,000(Certifications)
Where:
• Years = years of experience
• Degree = 1 if graduate degree, 0 otherwise
• Certifications = number of professional certifications
Multiple regression uses several predictors. Each coefficient (b₁, b₂, etc.) shows the effect of that variable while holding others constant.
From Line to Plane (and Beyond)
With 1 predictor, we fit a line. With 2 predictors, we fit a plane in 3D space. With more predictors, we fit a hyperplane in higher dimensions—same principle, just harder to visualize.
Adjusted R-Squared
Regular R² always goes up with more predictors. Adjusted R² penalizes for model complexity—it only increases if the new variable genuinely improves prediction. Always report adjusted R² for multiple regression.
Partial Effects
Holding Other Variables Constant
Interpretation of b₁ = 2,500:
"Among people with the same degree status, each additional year of experience is associated with $2,500 higher salary."
Each coefficient shows the effect of that variable while holding all other variables constant. This lets us isolate the unique contribution of each predictor.
Confounding and Control
b = 0.8 (significant)
Looks like ice cream causes drowning!
Ice cream effect: b = 0.1 (n.s.)
Temperature effect: b = 0.7 (sig.)
Temperature was the confounder!
Multiple regression helps identify confounders. When we control for temperature, the ice cream-drowning relationship disappears—temperature was driving both. This is statistical 'control.'
Testing Individual Predictors
Hₐ: bᵢ ≠ 0 (significant unique contribution)
t = bᵢ / SE(bᵢ)
| Predictor | b | SE | t | p |
|---|---|---|---|---|
| Years | 2500 | 400 | 6.25 | <.001 |
| Degree | 5000 | 1200 | 4.17 | <.001 |
Each coefficient has its own t-test. A significant p-value means that predictor contributes unique information beyond the other predictors. Non-significant doesn't mean unimportant—could be redundant with others.
Multicollinearity
When Predictors Are Correlated
When predictors are highly correlated, it's hard to separate their individual effects. The model can't tell which variable is responsible for the outcome.
Signs of Multicollinearity
Coefficients become imprecise
Small data changes → big coefficient changes
Coefficients opposite to expected direction
Model fits well but can't identify which variable matters
Symptoms include inflated standard errors, unstable coefficients, and counterintuitive signs. The overall model may fit well (high R²), but individual predictors aren't significant.
Variance Inflation Factor
VIF measures how much a coefficient's variance is inflated due to collinearity. VIF = 1 means no correlation with other predictors. VIF > 10 indicates severe multicollinearity.
Dealing with Multicollinearity
Drop one of the correlated predictors
Create composite score (e.g., average)
Helps with interaction terms
Use principal components as predictors
Options include removing redundant predictors, combining them into composites, centering (especially for interactions), or using PCA to create uncorrelated predictors.
Model Building
Variable Selection Methods
Automated methods can help select variables, but they have limitations. They can capitalize on chance and find 'best' models that don't replicate. Theory-driven selection is often preferred.
Comparing Models
| Criterion | Formula | Lower = Better? |
|---|---|---|
| Adjusted R² | Penalizes for k | No (higher better) |
| AIC | -2ln(L) + 2k | Yes |
| BIC | -2ln(L) + k·ln(n) | Yes |
| Mallows' Cp | SSE/MSE + 2k - n | Yes (close to k) |
Information criteria (AIC, BIC) balance fit with parsimony. BIC penalizes complexity more heavily than AIC. Compare models with the same outcome variable; lower AIC/BIC = better.
Model Building Guidelines
Good practice: use theory to guide variable selection, ensure adequate sample size, check all assumptions, validate on new data, and report the full model (not just significant predictors).
Real-World Examples
Apply your knowledge to practical scenarios
Predicting Employee Salaries
HR builds a salary prediction model: Salary = 30,000 + 2,000(Experience) + 8,000(Degree) + 500(Training). Where Experience is in years, Degree = 1 if graduate degree (0 otherwise), Training = number of certifications. Adjusted R² = 0.72.
Sample Data
{
"n": 200,
"predictors": 3,
"adj_r_squared": 0.72
}Key Insights
- Multiple regression coefficients are 'partial' effects—controlling for other variables
- Coefficients often change between simple and multiple regression
- Regression doesn't prove causation—unmeasured confounders may exist
Unmasking the Ice Cream-Drowning Confounder
Simple regression shows Ice Cream Sales significantly predicts Drowning Deaths (b = 0.8, p < 0.01). But when we add Temperature to the model, the results change dramatically.
Sample Data
{
"simple_model": {
"ice_cream_b": 0.8,
"ice_cream_p": 0.001
},
"multiple_model": {
"ice_cream_b": 0.05,
"ice_cream_p": 0.72,
"temperature_b": 1.2,
"temperature_p": 0.001
}
}Key Insights
- Adding controls can make spurious relationships disappear
- Multiple regression helps identify true predictors vs. confounders
- Always consider: 'What third variable could explain this?'
The Multicollinearity Trap
A researcher predicts test scores from Hours_Studied and Practice_Problems. Both are strongly correlated with test scores individually (r = 0.7 each), but in the multiple regression model, neither is significant (p > 0.05 for both). R² = 0.52.
Key Insights
- Multicollinearity: correlated predictors make individual coefficients unreliable
- Symptoms: high R² but non-significant predictors, large SEs, unstable coefficients
- VIF > 10 indicates serious multicollinearity problems
Multiple Regression Quiz
Test your understanding • Passing score: 70%