When Linear Isn't Enough
Curved Relationships
Not all relationships are linear. Growth often slows (diminishing returns), learning curves plateau, and physical processes can be exponential or logarithmic. Forcing a line on curved data gives poor predictions.
Spotting Non-Linearity
A curved pattern in residuals is the clearest sign of non-linearity. The scatter plot may also show the curve directly, but residual plots magnify the pattern.
Common Non-Linear Forms
Different shapes suit different phenomena. Logarithmic: diminishing returns. Exponential: compound growth. Quadratic: parabolic (max/min). Choose based on theory and data pattern.
Polynomial Regression
Adding Polynomial Terms
Polynomial regression adds powers of X (x², x³, etc.). Despite the curves, it's still 'linear regression' because the model is linear in the parameters (b₀, b₁, etc.).
Fitting Polynomial Models
The quadratic model captures the U-shape far better than the linear model. R² jumps from 0.45 to 0.94. But be careful—higher-degree polynomials aren't always better.
Understanding Polynomial Coefficients
Where x = hours of practice
• Intercept (10): Baseline performance
• x coefficient (5): Initial rate of improvement
• x² coefficient (-0.5): How quickly improvement slows
Peak: Maximum at x = -b₁/(2b₂) = -5/(2×-0.5) = 5 hours
With polynomial terms, individual coefficients are harder to interpret. Focus on the overall shape: where does it peak/trough? How steep is the curve? Often the shape matters more than individual coefficients.
Polynomial Pitfalls
Polynomials explode outside the data range
High-degree polynomials fit noise, not signal
x, x², x³ are highly correlated
Center x first, use lowest degree needed
Polynomial regression has pitfalls: wild behavior outside data range, tendency to overfit, and multicollinearity among polynomial terms. Center variables and use the lowest degree that fits well.
Model Selection Criteria
Fit vs Complexity
Training error always decreases with complexity. Test error decreases then increases (overfitting). Model selection criteria try to find the sweet spot—good fit without unnecessary complexity.
Information Criteria
• Prefers complex models
• Better for prediction
• Prefers simpler models
• Better for explanation
AIC and BIC balance fit (-2ln(L)) with a complexity penalty. BIC penalizes more heavily for large samples. Use AIC for prediction; BIC when you want to identify the 'true' model.
Testing on Held-Out Data
Cross-validation rotates which data is held out for testing. Average the test errors across folds. This gives an honest estimate of how well the model will predict new data.
Comparing Nested Models
Model 1: y = b₀ + b₁x
Model 2: y = b₀ + b₁x + b₂x²
Model 1 is nested in Model 2 (just set b₂ = 0)
H₀: Extra parameters = 0 (simpler model sufficient)
Hₐ: Extra parameters ≠ 0 (complex model better)
F = [(SSE₁ - SSE₂)/(df₁ - df₂)] / [SSE₂/df₂]
For nested models, use an F-test to see if the complex model is significantly better. If p > 0.05, the simpler model is preferred—the extra terms don't help enough.
Overfitting
Fitting Noise, Not Signal
Overfitting means the model captures noise (random variation) rather than just the underlying pattern. It fits training data perfectly but fails on new data.
Detecting Overfitting
Big gap between training and test performance
More predictors than needed, especially relative to n
Small changes in data → big changes in model
Extreme predictions for new observations
Key sign: big gap between training and test performance. Also watch for too many parameters relative to sample size, unstable coefficients, or wild predictions on new data.
How to Avoid Overfitting
More data = harder to overfit
Simpler models generalize better
Test on held-out data
Ridge/LASSO penalize complexity
Prevent overfitting by: getting more data, using simpler models, validating on held-out data, and using regularization techniques (Ridge, LASSO) that penalize model complexity.
The Bias-Variance Trade-off
• Misses patterns
• Consistent but wrong
• Fits noise
• Changes with sample
Simple models have high bias (consistently wrong) but low variance. Complex models have low bias but high variance (unstable). The best model balances both.
Real-World Examples
Apply your knowledge to practical scenarios
Advertising Diminishing Returns
A marketing analyst models Sales vs Advertising spend. Linear model: R² = 0.68, residuals show U-shaped pattern. Quadratic model (adding Ads²): R² = 0.89, residuals now random. Equation: Sales = 50 + 12(Ads) - 0.3(Ads²).
Key Insights
- Curved residuals signal need for polynomial terms
- Quadratic terms capture diminishing returns or accelerating effects
- Never extrapolate polynomial models outside the data range
The Overfitting Disaster
A data scientist has 50 data points and fits three models: Linear (R² = 0.45), Quadratic (R² = 0.62), and 15th-degree polynomial (R² = 0.99). They proudly present the polynomial model.
Key Insights
- Training R² always increases with complexity—it's misleading
- Test error (or cross-validation) reveals true predictive ability
- The simplest adequate model usually generalizes best
Choosing Between Competing Models
A researcher compares four models predicting patient recovery time. Model A (2 predictors): AIC = 245, BIC = 250. Model B (5 predictors): AIC = 238, BIC = 252. Model C (8 predictors): AIC = 235, BIC = 260. Model D (3 predictors): AIC = 240, BIC = 247.
Sample Data
{
"models": [
{
"name": "A",
"predictors": 2,
"AIC": 245,
"BIC": 250
},
{
"name": "B",
"predictors": 5,
"AIC": 238,
"BIC": 252
},
{
"name": "C",
"predictors": 8,
"AIC": 235,
"BIC": 260
},
{
"name": "D",
"predictors": 3,
"AIC": 240,
"BIC": 247
}
]
}Key Insights
- AIC and BIC can recommend different models—neither is always right
- AIC: better for prediction | BIC: better for explanation
- When in doubt, prefer simpler models (parsimony principle)
Model Comparison Quiz
Test your understanding • Passing score: 70%