Module 18: Missing Data - Statistics Visualizer

Why Missing Data Matters

The Impact of Missing Data

Survey Response: Income by Education

ID	Education	Income	Status
1	College	$65,000	✓
2	High School	Missing	?
3	Graduate	$95,000	✓
4	College	Missing	?
5	High School	$35,000	✓

40% of income data is missing - what do we do?

Missing data is ubiquitous in real research. Surveys have non-response, sensors fail, participants drop out. How we handle missing data can dramatically affect our conclusions - bad choices lead to bias.

Missing data isn't just annoying - it can bias your results

What Can Go Wrong

Biased Estimates

If high-income people refuse to report income, mean income is underestimated

Reduced Power

Discarding incomplete cases wastes data and reduces sample size

Invalid Inference

Standard errors and p-values assume complete data

Non-representative Sample

Complete cases may not represent the full population

Missing data causes multiple problems: biased estimates if missingness is related to the variable itself, reduced power from smaller effective samples, and potentially invalid statistical tests.

The method you choose depends on WHY data is missing

How Much Missing Is Too Much?

< 5%
Usually OK

5-10%
Manageable

10-20%
Concerning

20-40%
Serious

> 40%
Critical

But pattern matters more than percentage!

While there's no universal threshold, <5% missing is usually trivial, 5-20% needs careful handling, and >40% is severe. But the pattern of missingness matters more than the percentage - 10% MNAR can be worse than 30% MCAR.

Pattern of missingness > amount of missingness

Missing Data Mechanisms

Missing Completely At Random

Best Case Scenario

Example: A survey form blows away in the wind. The missing data has nothing to do with the person or their answers.

MCAR is rare but ideal - complete cases are representative, and simple methods work.

Missing Completely At Random (MCAR) means the probability of missing data is unrelated to any observed or unobserved variables. Like a truly random coin flip deciding which data points are lost. Rare in practice but easiest to handle.

MCAR: Missingness is pure random chance

Missing At Random

Most Common Case

Age Group	Income Missing %
18-30	30%
31-50	15%
51-70	10%

Example: Younger people are less likely to report income - but within each age group, missingness doesn't depend on actual income.

MAR allows good imputation if you have the predictive variables (like age).

Missing At Random (MAR) means missingness can be predicted from OTHER observed variables, but not from the missing variable itself. If young people skip income questions, but rich and poor young people skip equally, it's MAR.

MAR: Missingness depends on observed (not missing) variables

Missing Not At Random

Problematic Case

💰

High Income

70% refuse

📊

Middle Income

30% refuse

💵

Low Income

10% refuse

Example: High earners refuse to report income more often than low earners. The missingness depends on the actual (missing) income value.

MNAR causes bias that can't be fixed with standard imputation!

Missing Not At Random (MNAR) means the probability of missing depends on the unobserved value itself. High-income people are more likely to skip income questions. This is the hardest to handle and causes unavoidable bias without special techniques.

MNAR: Missingness depends on the missing value itself - problematic!

Diagnosing Missingness Pattern

How to Assess Missing Data Mechanism

Little's MCAR Test

Tests if missingness is completely random. Significant p = not MCAR.

Compare Groups

Compare complete vs. incomplete cases on other variables.

Logistic Regression

Predict missingness from observed variables (tests MAR).

Theory/Domain Knowledge

MNAR usually requires theoretical reasoning - it's often untestable.

You can test for MCAR using Little's test. Comparing complete cases to incomplete cases on observed variables helps assess MAR. MNAR is hardest to detect - it requires domain knowledge about why data is missing.

MNAR is usually assumed from domain knowledge, not tested statistically

Simple Approaches

Complete Case Analysis

Listwise Deletion (Complete Cases Only)

ID	X	Y	Z	Status
1	5	3	8	✓ Kept
2	3	NA	6	✗ Deleted
3	7	9	4	✓ Kept
4	NA	5	2	✗ Deleted

Pros: Simple, unbiased if MCAR

Cons: Wastes data, biased if not MCAR

Listwise deletion removes any case with any missing value. It's simple and gives unbiased estimates ONLY if data is MCAR. But it can waste enormous amounts of data - 10 variables with 5% each missing could lose 40% of cases!

Listwise deletion: simple but often wasteful

Available Case Analysis

Pairwise Deletion (Use Available Data)

Correlation matrix using all available pairs:

	X	Y	Z
X	1.00	n=85	n=92
Y	n=85	1.00	n=78
Z	n=92	n=78	1.00

Problem: Different n for each correlation - can lead to impossible matrices!

Pairwise deletion uses all available data for each specific analysis. For correlations, it uses all cases with both variables present. It retains more data but can cause inconsistent results and sometimes impossible covariance matrices.

Pairwise uses more data but can create statistical problems

Mean Imputation

Mean Imputation: Replace Missing with Mean

Before:

24NA68

After (mean=5):

24568

Problems:
• Underestimates variance (values artificially cluster at mean)
• Distorts correlations (pushes toward zero)
• Biased if not MCAR

Mean imputation replaces missing values with the variable's mean. While simple, it seriously underestimates variance (all imputed values are identical) and attenuates correlations. Generally NOT recommended except for trivial missing rates.

Mean imputation is simple but causes serious bias - avoid!

Multiple Imputation

The Gold Standard

Modern Best Practice

Dataset 1
NA→4.2

Dataset 2
NA→5.1

Dataset 3
NA→4.7

Dataset 4
NA→3.9

Dataset 5
NA→5.3

Each dataset gets different imputed values → analyze all → combine results

Multiple Imputation (MI) creates multiple complete datasets with different plausible imputed values, analyzes each, then pools results. This properly accounts for uncertainty about the missing values. It's the recommended approach under MAR.

Multiple imputation accounts for uncertainty in missing values

Three Steps of MI

Impute

Create m (usually 5-20) complete datasets with different imputed values based on a statistical model

Analyze

Run your desired analysis (regression, t-test, etc.) on each of the m datasets

Pool

Combine results using Rubin's rules: average estimates, combine variances (within + between)

MI involves three phases: (1) Imputation - generate multiple completed datasets using predictive models, (2) Analysis - perform your planned analysis on each dataset separately, (3) Pooling - combine results using Rubin's rules to get final estimates and proper standard errors.

Impute → Analyze → Pool

Imputation Algorithms

MICE (Chained Equations)

Imputes each variable using a regression model with all other variables as predictors. Most flexible - handles mixed data types.

Multivariate Normal

Assumes all variables are jointly normal. Faster but limited to continuous data.

Predictive Mean Matching

Imputes with observed values from similar cases. Preserves distribution shape.

MICE (Multiple Imputation by Chained Equations) is most popular because it handles mixed variable types and doesn't require joint normality. Predictive Mean Matching is useful for preserving realistic distributions by only imputing observed values.

MICE is the most flexible and widely used approach

Rubin's Rules

Combining Results (m = 5 imputations)

Pooled Estimate:

Q̄ = (Q₁+Q₂+...+Qₘ)/m

Total Variance:

T = W̄ + (1+1/m)B

W̄ = Average within-imputation variance
B = Between-imputation variance
T = Total variance (accounts for imputation uncertainty)

Rubin's rules combine estimates by averaging across imputations. The standard error combines within-imputation variance (uncertainty in data) and between-imputation variance (uncertainty from missing data). This gives proper inference.

Rubin's rules give correct standard errors and p-values

Other Modern Methods

Full Information Maximum Likelihood

Direct Estimation Approach

Instead of imputing, FIML estimates model parameters directly using all available data.

Advantages:
• Single-step (no separate imputation)
• Efficient (uses all data)
• Standard in SEM software

Limitations:
• Only for specific models
• Requires specialized software
• Harder to include auxiliary variables

FIML uses the likelihood function to estimate parameters using all available data without imputation. It's mathematically equivalent to MI under MAR and is built into most SEM software. Excellent choice when available.

FIML: Efficient, single-step estimation using all available data

Machine Learning Imputation

Modern ML Approaches

Random Forest Imputation

missForest: Uses random forests to predict missing values. Handles non-linear relationships, mixed data.

k-Nearest Neighbors

Impute from similar complete cases. Simple, intuitive, but sensitive to k and distance metric.

Deep Learning (Autoencoders)

Neural networks that learn to reconstruct data. Powerful for complex patterns but requires large data.

Machine learning methods like Random Forests and kNN can capture complex, non-linear relationships when imputing. They often predict well but may not properly reflect uncertainty unless combined with multiple imputation.

ML methods are powerful but need careful implementation for inference

Sensitivity Analysis for MNAR

When MNAR Is Suspected

Test how conclusions change under different MNAR scenarios:

Scenario	Assumption	Effect Estimate
MAR	Missing unrelated to outcome	β = 0.45
MNAR (mild)	Missing have 10% lower values	β = 0.38
MNAR (moderate)	Missing have 20% lower values	β = 0.32

If conclusions are robust across plausible MNAR scenarios, findings are more trustworthy.

When MNAR is suspected, sensitivity analysis tests how results change under different assumptions about the missing data mechanism. Pattern-mixture models and selection models are formal approaches to MNAR.

Sensitivity analysis shows how robust findings are to MNAR

★

Real-World Examples

Apply your knowledge to practical scenarios

Clinical Trial Dropout

In a 12-week depression treatment trial, 30% of participants drop out. Patients who aren't improving are more likely to drop out (MNAR suspected). Complete case analysis shows the treatment works great - but this may be because only responders remained.

Sample Data

{
  "total_enrolled": 200,
  "completed": 140,
  "dropout_rate": "30%",
  "dropout_pattern": "Higher in non-responders",
  "suspected_mechanism": "MNAR"
}

Q Why is complete case analysis problematic here? +

A If non-responders drop out more, analyzing only completers overestimates treatment effect. The 'successful' sample doesn't represent all patients who would receive the treatment.

Q What approach would be more appropriate? +

A Use intention-to-treat analysis with multiple imputation (conservative MAR assumption), plus sensitivity analysis varying MNAR assumptions. Pattern-mixture models can explicitly model the dropout mechanism.

Key Insights

Clinical trial dropout is often MNAR - worse outcomes lead to dropout
Intention-to-treat analysis requires handling missing data properly
Sensitivity analysis is crucial when MNAR is plausible

Income Non-Response in Census

A national survey has 25% missing income data. Analysis shows: higher education → more likely to skip income question, BUT within education levels, missingness doesn't vary by other observed characteristics. This suggests MAR.

Sample Data

{
  "total_respondents": 50000,
  "income_missing": "25%",
  "education_correlation": "Higher education → more missing",
  "mechanism_assessment": "Likely MAR given education"
}

Q How would you assess whether this is MAR vs MNAR? +

A Compare complete vs. incomplete cases on all observed variables. If education explains missingness but conditional on education, no other observed variable predicts it, MAR is plausible. MNAR would require income itself to predict missingness.

Q What imputation strategy would you recommend? +

A Use multiple imputation (MICE) including education and all other relevant predictors in the imputation model. Include auxiliary variables that predict income even if not in the final analysis model.

Key Insights

MAR is plausible when observed variables explain missingness
Include auxiliary variables in imputation models
Multiple imputation is preferred for MAR data

IoT Sensor Failures

Temperature sensors in a warehouse occasionally fail, producing missing readings. Analysis shows failures are random - they occur equally across all temperature ranges and times. This is true MCAR.

Sample Data

{
  "sensors": 50,
  "readings_per_day": 1440,
  "missing_rate": "3%",
  "pattern": "Random failures unrelated to temperature",
  "mechanism": "MCAR"
}

Q Why is this situation ideal for analysis? +

A MCAR means complete case analysis is unbiased - the observed readings are a representative random sample. Even simple methods like mean imputation would introduce minimal bias (though still underestimate variance).

Q What's the best approach for this data? +

A With only 3% MCAR missing, listwise deletion is fine for most analyses. For time series continuity, interpolation from neighboring time points is appropriate since sensor failures are brief.

Key Insights

True MCAR is rare but ideal when it occurs
Low MCAR rates (< 5%) are usually ignorable
Domain knowledge helps identify the missing mechanism

← Logistic Regression Next: Bayes' Theorem →

Missing Data

The Missing Data Problem

MCAR

MAR

MNAR

Imputation

Why Missing Data Matters

The Impact of Missing Data

What Can Go Wrong

How Much Missing Is Too Much?

Missing Data Mechanisms

Missing Completely At Random

Missing At Random

Missing Not At Random

Diagnosing Missingness Pattern

Simple Approaches

Complete Case Analysis

Available Case Analysis

Mean Imputation

Multiple Imputation

The Gold Standard

Three Steps of MI

Imputation Algorithms

Rubin's Rules

Other Modern Methods

Full Information Maximum Likelihood

Machine Learning Imputation

Sensitivity Analysis for MNAR

Real-World Examples

Clinical Trial Dropout

Sample Data

Key Insights

Income Non-Response in Census

Sample Data

Key Insights

IoT Sensor Failures

Sample Data

Key Insights

Missing Data Quiz

Quiz Results