Why Reduce Dimensions?
The Curse of Dimensionality
High-dimensional data (many variables) creates problems: data becomes sparse, distances lose meaning, models overfit, computation slows, and visualization is impossible beyond 3D. Dimensionality reduction compresses data while preserving the important structure.
The Core Insight
PC1 captures most of the variance.
Variables are often correlated - when one goes up, others tend to go up or down predictably. This means the data actually lies in a lower-dimensional space. PCA finds that space by identifying directions (principal components) that capture the most variance.
Where PCA Helps
PCA is a Swiss army knife: visualize high-D data in 2D, reduce features before ML, compress images, create uncorrelated variables, remove noise, fix multicollinearity. It's often the first step in exploring complex datasets.
How PCA Works
The PCA Algorithm
PCA finds orthogonal directions that maximize variance. Mathematically: compute the covariance matrix, find its eigenvalues and eigenvectors. Eigenvectors are the principal component directions; eigenvalues indicate how much variance each captures. Sort by eigenvalue, keep the top k.
Geometric View
PC2 is perpendicular, captures remaining variance.
Think of data as an ellipsoid cloud. PC1 is the longest axis - the direction of greatest spread. PC2 is perpendicular to PC1 and is the next longest. Each subsequent PC is perpendicular to all previous ones, capturing decreasing amounts of variance.
Why Standardize First
PCA maximizes variance, so variables with larger scales dominate. Income in dollars has more variance than height in centimeters just due to units, not importance. Standardizing (z-scores) puts all variables on equal footing. Always standardize unless variables are already comparable.
What PCA Produces
PCA outputs: (1) Eigenvalues showing variance per component, (2) Loadings showing how original variables combine into PCs, (3) Scores - each observation's values on the new PC axes. Use eigenvalues to choose k, loadings to interpret PCs, scores for downstream analysis.
Variance Explained
How Much Information Kept?
Each PC captures a portion of total variance. PC1 captures the most, PC2 the second most, etc. The percentage is the eigenvalue divided by sum of all eigenvalues. Keeping PC1 and PC2 (70% in this example) means 70% of the original information is preserved.
The Elbow Method for PCA
The scree plot shows eigenvalues (or variance explained) for each PC. The curve typically drops steeply then flattens. The 'elbow' where it bends suggests how many PCs to keep. Components after the elbow add little information but add noise and complexity.
How Many Components?
Several rules exist: keep enough for 80-95% variance, keep eigenvalues > 1 (Kaiser), use scree plot elbow, or cross-validate based on downstream performance. No single rule is best - use multiple and consider your goals.
Interpreting Components
What Variables Contribute?
| Variable | Loading | Interpretation |
|---|---|---|
| Income | +0.82 | Strong positive |
| Education | +0.75 | Strong positive |
| Age | +0.25 | Weak positive |
| Debt | -0.60 | Strong negative |
Loadings show how much each original variable contributes to a PC. High absolute loadings indicate important variables. Same-sign loadings move together; opposite signs move inversely. Use loadings to name/interpret components - here PC1 captures 'socioeconomic status'.
Visualizing Loadings and Scores
A biplot shows both scores (observations as points) and loadings (variables as arrows) on the same plot. Arrow direction shows loading signs; length shows magnitude. Points near an arrow have high values on that variable. Arrows pointing same direction indicate correlation.
Varimax and Other Rotations
.50, .45, .60, .30
.10, .15, .88, .82
Raw PCA loadings can be hard to interpret - variables load on multiple components. Rotation (like varimax) redistributes loadings to push them toward 0 or ±1, making each component depend on fewer variables. Doesn't change total variance explained, just makes interpretation easier.
PCA in Practice
Step-by-Step PCA
A typical PCA workflow: handle missing data, standardize variables, run PCA, examine scree plot to choose k, interpret loadings to understand components, then use scores for downstream tasks (visualization, clustering, regression). Document your choices.
When Does PCA Work?
PCA assumes linear relationships - it finds linear combinations. Non-linear patterns need kernel PCA. Outliers distort results - robust PCA helps. Binary/categorical data may need different methods. More observations than variables is ideal, though not strictly required.
Common Mistakes
Common errors: not standardizing lets large-scale variables dominate; keeping too few/many PCs either loses signal or keeps noise; over-interpreting weak loadings; not handling outliers. PCA shows correlation structure - don't claim causal relationships from it.
Beyond PCA
PCA alternatives: Factor Analysis models underlying latent variables. t-SNE and UMAP capture non-linear structure for visualization. ICA finds independent sources. Autoencoders use neural networks for non-linear reduction. Choose based on your goals and data characteristics.
Real-World Examples
Apply your knowledge to practical scenarios
Eigenfaces for Face Recognition
A face recognition system has a database of 10,000 face images, each 100×100 pixels (10,000 dimensions). PCA reduces each face to 150 principal components while retaining 95% of variance.
Key Insights
- PCA enables high-dimensional pattern recognition by reducing noise
- Principal components can have interpretable meanings (eigenfaces)
- Dimensionality reduction often IMPROVES downstream performance
Portfolio Risk Factor Analysis
A fund manager tracks returns of 500 stocks. PCA on the correlation matrix reveals: PC1 explains 40% of variance, PC2 explains 15%, PC3 explains 8%. The first 10 PCs explain 80% of total variance.
Key Insights
- Stock returns are driven by a few systematic factors, not 500 independent variables
- PC1 in finance is typically the market factor
- PCA reveals the true dimensionality of portfolio risk
Gene Expression Analysis
A cancer researcher measures expression of 20,000 genes in 100 tumor samples. PCA visualization shows samples naturally clustering into 3 groups in PC1-PC2 space, corresponding to different cancer subtypes.
Key Insights
- PCA handles p >> n situations where most methods fail
- Loadings reveal which original variables drive each PC
- Visualization in PC space can reveal natural groupings
PCA Quiz
Test your understanding • Passing score: 70%