Why Logistic Regression?
Binary Outcomes Don't Fit Lines
With binary outcomes (0 or 1), linear regression fails: it can predict impossible values like -0.3 or 1.5. We need a model that outputs probabilities between 0 and 1.
The S-Curve Solution
The logistic (S-shaped) curve naturally stays between 0 and 1. It asymptotes to 0 and 1 but never crosses. The output is interpreted as P(Y=1)—probability of success.
When to Use Logistic Regression
Disease present/absent
Survival/death
Treatment response
Purchase yes/no
Churn yes/no
Fraud yes/no
Pass/fail
Graduate/dropout
Admit/reject
Success/failure
Present/absent
Yes/no responses
Use logistic regression whenever your outcome is binary (two categories). It's one of the most widely used methods in medicine, business, and social sciences.
The Logistic Function
The Math Behind the S-Curve
• When z → -∞, P → 0
• When z = 0, P = 0.5
• When z → +∞, P → 1
The logistic function transforms the linear combination (b₀ + b₁X) into a probability. No matter how large or small the input, output stays between 0 and 1.
The Logit Transformation
The logit (log-odds) transformation linearizes the relationship. Logistic regression predicts log-odds as a linear function of X. This is why it's called 'linear' even though the probability curve is S-shaped.
What Coefficients Mean
"For each 1-unit increase in X, the log-odds of Y=1 increase by 0.5"
Or equivalently:
"The odds multiply by e^0.5 = 1.65"
↑ X → ↑ P(Y=1)
X doesn't affect P
↑ X → ↓ P(Y=1)
Coefficients show change in log-odds per unit X. Positive b means higher X → higher probability of Y=1. The magnitude tells you how strongly X affects the outcome.
Odds Ratios
Understanding Odds
Odds express probability differently. If P = 0.75, odds = 3 ("3 to 1 for"). Odds range from 0 to ∞. Odds = 1 means 50-50 chance. Odds > 1 means more likely than not.
Comparing Groups
b = 1.6, so OR = e^1.6 = 4.95
Interpretation:
"Smokers have ~5× the odds of lung cancer compared to non-smokers"
Protective
No effect
Risk factor
The odds ratio (OR) = e^b. OR = 5 means the odds are 5 times higher. OR = 0.5 means odds are halved. OR = 1 means no effect. This is the standard way to interpret logistic coefficients.
Confidence Intervals for OR
95% CI for b: [1.6 - 0.59, 1.6 + 0.59] = [1.01, 2.19]
95% CI for OR: [e^1.01, e^2.19] = [2.75, 8.94]
Conclusion: OR is between 2.75 and 8.94
Since CI doesn't include 1, effect is significant
The confidence interval for OR tells you the range of plausible effect sizes. If the CI doesn't include 1, the effect is statistically significant. Wider CI = more uncertainty.
How to Report Results
"Logistic regression was used to predict heart disease from cholesterol level. Higher cholesterol significantly increased the odds of heart disease (OR = 1.08, 95% CI [1.03, 1.13], p = .001). For each 10 mg/dL increase in cholesterol, the odds of heart disease increased by 8%. The model correctly classified 73% of cases (AUC = 0.79)."
Report: OR with 95% CI, p-value, and direction of effect. Also report model fit (classification accuracy, AUC). ORs are much more interpretable than raw coefficients.
Model Evaluation
The Confusion Matrix
| Predicted | ||
| Positive | Negative | |
| Actual Positive | TP True Positive | FN False Negative |
| Actual Negative | FP False Positive | TN True Negative |
The confusion matrix shows how predictions compare to reality. TP and TN are correct; FP (false alarm) and FN (missed detection) are errors. This is the foundation for many metrics.
Key Performance Metrics
Accuracy is overall correctness. Sensitivity (recall) is how well you catch positives. Specificity is how well you correctly identify negatives. The right metric depends on your goals.
ROC Curve and AUC
The ROC curve plots sensitivity vs (1-specificity) at different thresholds. AUC (area under curve) summarizes performance: 0.5 is random guessing, 1.0 is perfect classification.
Choosing the Cutoff
• More positives predicted
• Higher sensitivity
• More false positives
• Use when missing cases is costly
• Fewer positives predicted
• Higher specificity
• More false negatives
• Use when false alarms are costly
The default 0.5 threshold isn't always optimal. Lower it if missing positives is costly (disease screening). Raise it if false positives are costly (criminal conviction). Choose based on costs.
Real-World Examples
Apply your knowledge to practical scenarios
Predicting Customer Churn
A telecom company builds a logistic regression to predict customer churn. Model: logit(P) = -2.5 + 0.8(Complaints) + 0.05(MonthsWithService) - 0.3(SatisfactionScore). AUC = 0.82.
Sample Data
{
"n": 5000,
"churn_rate": 0.15,
"AUC": 0.82
}Key Insights
- Convert coefficients to odds ratios for interpretation: OR = e^b
- Negative coefficients mean the factor REDUCES the odds of the outcome
- AUC measures how well the model ranks cases—0.82 is good
Disease Screening Trade-offs
A hospital develops a logistic model to screen for a rare disease (prevalence 2%). At the default 0.5 cutoff: Sensitivity = 60%, Specificity = 95%. At a 0.2 cutoff: Sensitivity = 90%, Specificity = 80%.
Sample Data
{
"prevalence": 0.02,
"cutoff_05": {
"sensitivity": 0.6,
"specificity": 0.95
},
"cutoff_02": {
"sensitivity": 0.9,
"specificity": 0.8
}
}Key Insights
- Threshold choice depends on costs: lower for serious conditions where missing cases is costly
- Sensitivity and specificity don't tell the whole story—PPV depends on prevalence
- Rare diseases: even 'accurate' tests have low positive predictive value
Credit Card Fraud Detection
A bank builds a fraud detection model. Fraud rate is 0.1% (rare!). At default 0.5 threshold: catches 40% of fraud, 99.9% specificity. They're considering lowering threshold but are concerned about customer complaints from false positives.
Key Insights
- With imbalanced classes, accuracy is misleading
- Threshold selection requires explicit cost-benefit analysis
- Rare events make false positives inevitable in absolute numbers
Logistic Regression Quiz
Test your understanding • Passing score: 70%