Dimensionality reduction

Why this matters

Real datasets often have dozens or hundreds of columns that move together: height, arm span and shoe size; the prices of related stocks; thousands of gene-expression levels driven by a few biological processes. Dimensionality reduction replaces many correlated columns with a few new ones that keep most of the information. That lets you plot high-dimensional data in 2D, speed up models, reduce noise, and see the hidden factors in the data. Principal component analysis (PCA) is the standard first tool.

The idea

PCA finds new axes, called principal components, for the data:

1. The first component is the direction along which the data varies the most.
2. The second is the direction of most remaining variance, at a right angle (orthogonal) to the first.
3. And so on, up to the number of original features.

Projecting the data onto the first few components gives a compact version that keeps as much spread (variance) as possible. If five measurements are really driven by two hidden factors, the first two components will capture almost all the variance.

The linear algebra

This is where your linear algebra pays off. For a data matrix X with n rows:

1. centre:      Xc = X - column means
2. covariance:  C  = Xc.T @ Xc / (n - 1)          (features x features)
3. eigen:       C v = lambda v
4. sort eigenvalues from largest to smallest; the eigenvectors are the components
5. project:     Z = Xc @ V_k                      (n x k)

Each eigenvalue is the variance along its component, so the explained variance ratio of component i is lambda_i / sum(lambda).

import numpy as np

Xc = X - X.mean(axis=0)
cov = Xc.T @ Xc / (len(X) - 1)
values, vectors = np.linalg.eigh(cov)        # eigh: for symmetric matrices, ascending order
order = np.argsort(values)[::-1]
values, vectors = values[order], vectors[:, order]
ratio = values / values.sum()

Libraries usually compute the same thing with the singular value decomposition (SVD) of Xc, which is numerically more stable: the squared singular values divided by n - 1 are the eigenvalues.

Worked example: two correlated columns

Suppose height and arm span (in cm) have variances 100 and 110 and covariance 100, an almost perfect correlation of about 0.95. The covariance matrix's eigenvalues are about 205 and 5. The first component, roughly (0.69, 0.73), is "overall body size" and explains 205 / 210 ≈ 97.6% of the variance. The second, roughly (-0.73, 0.69), is "arms long for your height" and explains the remaining 2.4%. Keeping one component loses very little.

How many components?

• Plot the explained variance ratio for each component (a scree plot) and look for the elbow.
• Or keep enough components to reach a target such as 90% or 95% of cumulative variance.
• For visualisation, use 2 (or 3), whatever they explain, and report the percentage on the axes.

Scale first (usually)

PCA chases variance, so columns in big units dominate, just as in k-means. If columns are in different units (dollars, years, counts), standardise them first; this is PCA on the correlation matrix. If they share units and their scale is meaningful (pixel intensities, expression levels already normalised), you may centre only.

Reading the components

Each component is a weighted combination of the original features. The weights are called loadings. A component with large positive loadings on all the size measurements is a "size" factor. Eigenvector signs are arbitrary: v and -v are equally valid, so do not read meaning into the sign alone, and compare components across runs only up to sign.

In scikit-learn

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

pipe = make_pipeline(StandardScaler(), PCA(n_components=0.95))   # keep 95% of variance
Z_train = pipe.fit_transform(X_train)
Z_test = pipe.transform(X_test)              # fitted on train only
print(pipe[-1].explained_variance_ratio_)

For plotting complex, non-linear structure, people also use t-SNE and UMAP. They make good pictures but distort distances, so use them for exploration, not as model features.

Common mistakes

• Forgetting to centre, which makes the first component point at the mean instead of the spread.
• Not scaling mixed-unit features, so PCA just finds the largest-unit column.
• Fitting PCA on all data before splitting, which leaks test information (fit on train, transform test).
• Expecting components to be causes. They are directions of variance, which may or may not match real factors.
• Discarding low-variance components that carry the signal. PCA ignores the label; a rare but important pattern can live in a small component.

Quick recap

• PCA finds orthogonal directions of maximum variance: the eigenvectors of the covariance matrix.
• Explained variance ratio = eigenvalue / total; keep enough components for your purpose.
• Centre always, standardise mixed units, fit on training data only.
• Loadings show what each component means; signs are arbitrary.

Try it

On the Do tab, implement PCA with np.linalg.eigh and show that two components capture almost all the variance of five measurements.