Dimension Reduction Taxonomy

In the era of big data, high-dimensionality is both a blessing and a curse. While rich feature spaces capture nuanced patterns, they also introduce the curse of dimensionality: data becomes sparse, distances lose their meaning, and computational costs skyrocket. Dimensionality reduction is the arsenal of techniques designed to navigate this paradox. It compresses data into lower-dimensional spaces while preserving its essential structure — enabling visualization, noise reduction, faster algorithms, and deeper insights. This article walks a comprehensive taxonomy of these methods, from the simplest statistical summaries to state-of-the-art manifold learning and deep neural architectures, covering over a dozen distinct families with exhaustive mathematical treatment, historical context, and practical implementation guidance.

What is Dimensionality Reduction?

At its core, dimensionality reduction is the process of reducing the number of random variables under consideration, obtaining a set of principal variables. It is broadly divided into feature selection (choosing a subset of the original features) and feature extraction (transforming the data into a new, lower-dimensional space). The fundamental trade-off is between information loss and structural gain. A successful reduction removes redundancy and noise, making the latent structure of the data explicit. The network in Fig 1 illustrates the high-level goal: mapping a high-dimensional cloud of points into a comprehensible 2D or 3D space.

This diagram captures the overarching goal of every technique discussed in this article. On the left, we see a high-dimensional space where points are scattered across many dimensions (often >3). These points may belong to different classes (colored clusters), but their structure is obscured by the curse of dimensionality. The arrow labeled "Dimension Reduction" represents the transformation applied by methods such as PCA, t-SNE, or autoencoders. On the right, the same points are projected into a lower-dimensional space (e.g., 2D). The clusters become visually distinct, preserving the essential relationships — which points are close, which are far, and how they group together. This visual clarity is the primary motivation for many reduction techniques: to reveal the latent structure hidden in high-dimensional data.

The mathematical backbone of most linear techniques is the eigenvalue decomposition or the singular value decomposition (SVD). Non-linear methods relax the assumption of linearity, often relying on graph-based distances or probabilistic frameworks. Before diving into the taxonomy, it is crucial to understand the foundational statistics that drive all these methods: measures of central tendency and spread.

Foundations: Measures of Central Tendency and Spread

Before projecting data, we often summarize it. The mean (average) and median are the simplest forms of reduction — they compress a vector of numbers into a single representative value. The variance and standard deviation quantify the spread. These are the building blocks of more sophisticated techniques like PCA, which maximizes variance, and robust statistics, which rely on the median.

1. Statistical Summaries: Mean, Median and Variance

The most fundamental form of dimensionality reduction is taking the simple average (arithmetic mean). When faced with a vector of numbers (e.g., a student's marks in different subjects), we collapse it to a single scalar. The median serves as a robust alternative. These methods reduce dimension from n to 1. They are parameter-free, interpretable, and computationally optimal.

For example, consider Student A with marks [90, 85, 88] in Mathematics, Physics, and Chemistry, and Student B with [70, 75, 80]. By averaging, Student A's profile reduces to 87.7, and Student B's to 75.0. This dramatic reduction from 3 dimensions to 1 allows for an immediate and intuitive comparison: Student A is outperforming Student B overall. The diagram in Fig 2 visualizes this exact process.

This figure illustrates the power of the arithmetic mean as a dimensionality reduction tool. Student A's scores are displayed as bars of varying heights: 90 in Math, 85 in Physics, and 88 in Chemistry. Each of these is a separate dimension. The arrow points to a single red circle containing the average, 87.7. Student B's scores (70, 75, 80) are similarly collapsed to a green circle with the value 75.0. By reducing each student's 3-dimensional profile to a 1-dimensional scalar, we can instantly compare the two students: A is performing better overall. This example demonstrates the fundamental trade-off of all reduction methods: we lose subject-specific information (e.g., Student B might be stronger in Chemistry than A, despite a lower overall average), but we gain a compact, comparable summary that is useful for ranking and decision-making.

1.1 Arithmetic Mean (Simple Average)

The arithmetic mean is defined as:

$$\mu =\frac{1}{n}\underset{i=1}{\overset{n}{\Sigma }}{x}_i$$

It minimizes the sum of squared errors. For the example in Fig 2, Student A's average is 87.7, and Student B's is 75.0. These single numbers capture the overall performance level of each student, allowing a direct comparison. In feature engineering, averages are used to normalize data (e.g., subtracting the mean) and to create aggregated features.

Mathematical properties: The mean is the minimum-variance unbiased estimator for the location parameter of a Gaussian distribution. It satisfies the linearity property: for constants a and b,

$$E[aX+b]=aE[X]+b$$

The mean is also the solution to the least-squares problem:

$$\underset{c}{argmin}\underset{i=1}{\overset{n}{\Sigma }}(x_i-c{)}^2$$

which is why it is the natural choice for minimizing reconstruction error in many statistical models. The mean is sensitive to every data point, which is both a strength (it uses all information) and a weakness (outliers can skew it). In practice, the mean is computed in O(n) time and is a fundamental building block for more complex algorithms like PCA, where data is centered by subtracting the mean.

1.2 Median

The median is the middle value when observations are ordered. For an odd number of observations, it is the central element; for an even number, it is the average of the two central elements. It minimizes the sum of absolute deviations. The median is robust; an outlier does not shift it. This makes the median invaluable in income data, sensor readings prone to glitches, and any domain where outliers must not distort the summary. The median absolute deviation (MAD), defined as

$$\mathrm{MAD}=\mathrm{median}(|x_i-\mathrm{median}(x)|)$$

is a robust scale estimator often used in feature filtering.

The median has a breakdown point of 50%, meaning that up to half of the data can be arbitrarily corrupted without changing the median. This property makes it the preferred summary for skewed distributions, such as household income or house prices. In high-dimensional settings, computing the median is more expensive than the mean, requiring O(n log n) time or O(n) with selection algorithms. However, for many robust applications, the median is indispensable, especially when combined with MAD for outlier detection and feature scaling.

1.3 Variance and Standard Deviation

While not a reduction in dimension on its own, variance measures the spread of data and is the objective function for Principal Component Analysis. Variance is defined as:

$${\sigma }^2=\frac{1}{n}\underset{i=1}{\overset{n}{\Sigma }}(x_i-\mu {)}^2$$

Feature selection often relies on the low variance filter: features with variance near zero are considered constant and are removed. This is a straightforward reduction step, effectively reducing the number of features by eliminating those that carry little information.

The standard deviation σ is the square root of the variance and has the same units as the original data, making it more interpretable. In practice, the variance is computed using the two-pass formula for numerical stability. The coefficient of variation (CV = σ/μ) is a normalized measure of dispersion useful for comparing features with different scales. Variance is the cornerstone of PCA, where we seek projections that maximize the retained variance, effectively reducing dimensionality while preserving the most informative directions of the data.

Applications

• Baseline forecasting (e.g., average sales per day).
• Robust outlier filtering (median absolute deviation).
• Feature selection via low-variance filtering.

Strengths and limitations

Strengths	Limitations
Extremely fast and interpretable.	Loss of all relational structure.
No parameters to tune.	Mean is highly sensitive to outliers.
The foundation for all advanced methods.	Median is expensive for very large datasets.

2. Matrix Factorization: PCA and SVD

Principal Component Analysis (PCA) and the Singular Value Decomposition (SVD) are the twin pillars of linear dimensionality reduction. They find linear combinations of the original features that capture the maximum variance. PCA identifies the directions (principal components) that maximize the variance of the projected data. SVD is a robust matrix decomposition that underpins PCA and provides a direct way to compute the components, especially for high-dimensional or sparse matrices.

This diagram illustrates the core mechanism of PCA. In the input space (the 2D plane), points are distributed with a clear elongated shape along the diagonal. PCA finds the direction of maximum variance, labeled PC1 (Principal Component 1), which is the green line. The dashed lines show the orthogonal projections of each point onto PC1. The projected points (green circles on the PC1 axis) are the 1D representation of the data. This projection preserves the spread of the data as much as possible; points that were far apart in the 2D space remain far apart on the PC1 axis. The variance along PC1 is the largest eigenvalue of the covariance matrix. This is why PCA is often used for feature extraction: it finds a new coordinate system that aligns with the inherent structure of the data, allowing us to discard dimensions with low variance (noise) and retain the most informative directions.

2.1 Mathematical Foundations

Given a centered data matrix X of size n × p, PCA seeks a set of k orthonormal vectors (principal components) that maximize the variance of the projected data. The first principal component w is the eigenvector of the covariance matrix X^TX corresponding to the largest eigenvalue. The objective is:

$$w^{*}=\underset{w}{argmax}{w}^T{X}^{T}Xw$$

subject to $||w||=1$.

The Singular Value Decomposition (SVD) of X is:

$$X=U\Sigma V^T$$

where the columns of V contain the principal components. The variance explained by the i-th component is:

$$\frac{{\sigma }_i^2}{\underset{j=1}{\overset{p}{\Sigma }}{\sigma }_j^2}$$

This connection makes PCA numerically stable and allows it to handle n << p.

Derivation of PCA via Lagrange multipliers: To find the first principal component, we maximize $w^{T}Cw$ subject to $w^{T}w=1$, where C = X^TX is the covariance matrix. The Lagrangian is:

$$L(w,\lambda )=w^{T}Cw-\lambda (w^{T}w-1)$$

Taking the derivative with respect to w yields:

$$\frac{\partial L}{\partial w}=2Cw-2\lambda w=0$$

which simplifies to the eigenvalue equation:

$$Cw=\lambda w$$

Thus, the principal components are the eigenvectors of C, and the variance explained by each is the corresponding eigenvalue λ. Subsequent components are found recursively by deflating the covariance matrix, ensuring orthogonality.

2.2 Variants of PCA

Kernel PCA extends PCA to non-linear spaces via the kernel trick, mapping data into a high-dimensional feature space using a kernel function κ(x, y). The kernel matrix K is computed and centered, and PCA is performed on K, allowing non-linear structure to be captured. Sparse PCA imposes sparsity on the loadings, improving interpretability by selecting a subset of original features. Robust PCA decomposes the matrix into low-rank and sparse components, resisting outliers and gross corruption. Incremental PCA updates components online as new data arrives, and Randomized PCA uses random projections to approximate the SVD for speed.

Each variant addresses a specific limitation of the vanilla PCA. Kernel PCA is powerful for datasets with non-linear manifolds, but it is computationally expensive for large datasets due to the O(n³) cost of eigendecomposition of the kernel matrix. Sparse PCA is favored in genomics and text analysis where interpretability is paramount. Robust PCA is the go-to for surveillance, face recognition, and any domain with severe outliers. Randomized PCA is now a standard preprocessing step for huge datasets, providing a fast approximation with controlled error.

2.3 Example: Student Performance Analysis

If we had grades in Mathematics, Physics, Chemistry, and Literature, PCA could reduce these 4 dimensions to 2 principal components. The first component might represent "Quantitative Ability" (high loadings on Math/Physics/Chem), while the second might represent "Verbal Ability" (high loading on Literature). Plotting students in this 2D space reveals clusters of "STEM-oriented" and "Humanities-oriented" students.

For instance, suppose the covariance matrix has eigenvectors with weights: for PC1, [0.6, 0.6, 0.5, 0.1] and for PC2, [0.1, 0.1, 0.2, 0.9]. A student with high grades in Math, Physics, and Chem but average in Literature will have a high PC1 score and low PC2 score, placing them on the STEM cluster. Conversely, a student with high Literature but lower science scores will have a low PC1 but high PC2. This 2D representation captures the essential academic profile of each student, enabling educators to identify strengths and tailor instruction, while also providing a compact visualization of the entire cohort.

Applications

• Image compression (eigenfaces).
• Genomic data analysis (population structure).
• Financial risk modeling (factor analysis).
• Anomaly detection via reconstruction error.

Strengths and limitations

Strengths	Limitations
Linear, fast, and globally optimal.	Assumes linear relationships.
Provides interpretable components.	Sensitive to scaling and outliers.
SVD is numerically robust.	Kernel PCA is expensive for large datasets.

3. Pooling: Max, Average and Global Pooling

Pooling is a core operation in signal processing and Convolutional Neural Networks (CNNs) that aggressively downsamples feature maps. It reduces the spatial dimensions while preserving the most salient information. It operates on local neighborhoods (e.g., 2×2 windows), sliding a window across the image and applying a statistic. This introduces translation invariance and drastically reduces the number of parameters in subsequent layers.

This figure demonstrates the mechanics of max pooling, a cornerstone of convolutional neural networks. On the left, a 4×4 grid of numbers represents a feature map (e.g., the output of a convolution layer). A 2×2 sliding window (stride 2) scans the grid. For each window, max pooling selects the largest number. For the top-left window (values 1, 3, 5, 7), the maximum is 7. For the top-right window (2, 4, 6, 8), the maximum is 8. For the bottom-left window (9, 11), the maximum is 11, and for the bottom-right window (10, 12), the maximum is 12. The result is a 2×2 output matrix containing only the most salient features of each region. This operation reduces the spatial dimensions by half (from 4×4 to 2×2), discarding 75% of the data. Crucially, max pooling provides translation invariance: if the input shifts slightly, the maximum value in each window often remains the same, making the network robust to small positional changes in the input image. This is why pooling is so effective for tasks like object recognition.

3.1 Max Pooling

Max Pooling selects the maximum value from each window. The mathematical operation is:

$$\mathrm{output}=\max (\mathrm{window})$$

Backpropagation routes the gradient only to the argmax position.

The popularity of max pooling stems from its ability to preserve the most discriminative feature in each local region, such as an edge or a texture pattern. The gradient flow is sparse, which can help with training stability. However, max pooling discards all other information in the window, which can be a disadvantage if multiple features are present. Despite this, it has been a default choice in architectures like AlexNet and VGG, and its simplicity makes it computationally efficient.

3.2 Average Pooling

Average Pooling computes the mean of the window. It smoothes the signal and suppresses noise. It is often used in the final layers of CNNs (Global Average Pooling) to collapse the entire feature map into a single vector per channel, drastically reducing parameters.

Average pooling is a linear operation and is equivalent to a convolution with a uniform kernel. The gradient is distributed uniformly across all neurons in the window, encouraging the network to learn smooth, distributed features. In modern architectures like ResNet and EfficientNet, global average pooling has largely replaced fully connected layers, acting as a strong regularizer that reduces overfitting and forces the network to learn holistic representations of the input. It is also used in segmentation networks to aggregate spatial information for classification of each pixel.

3.3 Global Pooling

Global Pooling applies the aggregation over the entire spatial dimension, leaving a 1D vector of length equal to the number of channels. This is the ultimate form of spatial dimension reduction.

Global pooling is a critical component in modern CNNs, often used before the final classification layer. It dramatically reduces the number of parameters, making the network more efficient and less prone to overfitting. Global max pooling selects the strongest activation for each channel, while global average pooling computes the average. Global average pooling has been shown to improve generalization and is a key design choice in architectures like ResNet and DenseNet. It also makes the network spatially invariant, allowing it to handle inputs of varying sizes.

Applications

• Object recognition in images.
• Reducing computational load in deep neural networks.
• Time-series downsampling.

Strengths and limitations

Strengths	Limitations
Parameter-free and computationally cheap.	Loss of precise positional information.
Provides translation invariance.	Chooses a fixed window size (prior knowledge required).
Max pooling captures strong activations well.	Discards potentially useful information.

4. Non-Linear Manifold Learning: t-SNE and UMAP

Linear methods like PCA fail when the data lies on a curved manifold. Non-linear techniques assume that high-dimensional data is embedded in a low-dimensional manifold. t-SNE and UMAP are the two most prominent modern algorithms for visualizing high-dimensional data in 2D or 3D. They excel at preserving local structure (neighborhoods) and are ubiquitous in exploratory data analysis for biology, NLP, and computer vision.

This figure shows the typical output of t-SNE or UMAP: a 2D scatter plot where distinct clusters are clearly separated. The three large ellipses (blue, orange, and green) represent groups of data points that are close to each other in the original high-dimensional space. For example, in a single-cell RNA-seq experiment, these clusters might correspond to different cell types (e.g., T-cells, B-cells, and natural killer cells). The two red points in the corners are outliers that are distinct from all clusters. The key insight is that the high-dimensional relationships (which points are neighbors) are preserved in the low-dimensional embedding. t-SNE and UMAP achieve this by converting distances into probabilities and then minimizing the divergence between the high-dimensional and low-dimensional probability distributions. This makes them invaluable for exploratory data analysis, where the goal is to visually identify patterns that are invisible in the original high-dimensional space. UMAP often produces tighter clusters and better preserves global distances than t-SNE, but both are sensitive to hyperparameters like perplexity or n_neighbors.

4.1 t-Distributed Stochastic Neighbor Embedding (t-SNE)

t-SNE converts high-dimensional Euclidean distances into conditional probabilities. The similarity of point j to point i is:

$$p_{j|i}=\frac{\exp (-||x_i-x_j{||}^2/2{\sigma }_i^2)}{\underset{k\ne i}{\Sigma }\exp (-||x_i-x_k{||}^2/2{\sigma }_i^2)}$$

In the low-dimensional space, t-SNE uses a Student-t distribution (with one degree of freedom):

$$q_{ij}=\frac{(1+||y_i-y_j{||}^2{)}^{-1}}{\underset{k\ne l}{\Sigma }(1+||y_k-y_l{||}^2{)}^{-1}}$$

It minimizes the Kullback-Leibler (KL) divergence:

$$\mathrm{KL}(P||Q)=\underset{i\ne j}{\Sigma }{p}_{ij}\log (\frac{p_{ij}}{q_{ij}})$$

The gradient of the KL divergence with respect to the low-dimensional coordinates y_i is:

$$\frac{\partial \mathrm{KL}}{\partial y_i}=4\underset{j}{\Sigma }(p_{ij}-q_{ij})(y_i-y_j)(1+||y_i-y_j{||}^2{)}^{-1}$$

The heavy-tailed Student-t distribution in the low-dimensional space alleviates the "crowding problem" by allowing points to be placed farther apart without incurring a large penalty. The perplexity parameter, which can be thought of as the effective number of neighbors, governs the balance between local and global structure. t-SNE is widely used for visualization, but it is computationally expensive for large datasets due to the O(n²) cost of computing all pairwise affinities. The Barnes-Hut approximation reduces this to O(n log n) and makes it feasible for datasets up to ~100,000 points.

4.2 Uniform Manifold Approximation and Projection (UMAP)

UMAP constructs a fuzzy simplicial complex representation of the data and finds a low-dimensional embedding with a similar structure. It minimizes a cross-entropy loss:

$$\underset{ij}{\Sigma }[p_{ij}\log (\frac{p_{ij}}{q_{ij}})+(1-p_{ij})\log (\frac{1-p_{ij}}{1-q_{ij}})]$$

UMAP is generally faster than t-SNE and better preserves global structure.

UMAP is grounded in Riemannian geometry and algebraic topology, giving it a stronger theoretical foundation than t-SNE. It assumes that the data is uniformly distributed on a Riemannian manifold and estimates the local metric using nearest-neighbor distances. The low-dimensional embedding is optimized using stochastic gradient descent, making it highly scalable. UMAP's key parameters, n_neighbors and min_dist, allow fine control over the preservation of local vs. global structure. It has become the default visualization tool for large biological datasets, such as single-cell RNA-seq with millions of cells, due to its speed and ability to retain meaningful global structure.

4.3 Key Differences and Examples

In single-cell RNA-seq data, t-SNE and UMAP are the de facto tools for identifying cell subtypes. UMAP often provides a more interpretable embedding, with clusters arranged to reflect developmental lineages, while t-SNE tends to separate clusters equally regardless of global relationships.

For example, in a dataset of hematopoietic stem cells differentiating into various blood cell types, UMAP often places progenitor cells in a central location with mature cell types radiating outward, reflecting the biological trajectory. t-SNE, on the other hand, may spread clusters apart without preserving the continuous transitions. This difference arises because t-SNE's cost function focuses on local similarities, while UMAP's cross-entropy loss balances local and global structure. In practice, both methods are used in conjunction: t-SNE for detailed cluster inspection and UMAP for a broader view of the data landscape. Researchers often run multiple perplexities or n_neighbors values to ensure the stability of their findings.

Applications

• Visualization of high-dimensional genomic data.
• Exploratory analysis of image embeddings.
• Anomaly detection.
• Preprocessing for clustering algorithms.

Strengths and limitations

Strengths	Limitations
Excellent at revealing local clusters.	Stochastic nature leads to non-deterministic results.
Handles non-linear manifolds effectively.	t-SNE can be very slow for large datasets.
UMAP preserves global structure better.	Sensitive to hyperparameters (perplexity, n_neighbors).

5. Deep Learning for Reduction: Autoencoders

Autoencoders are neural networks trained to replicate their input at the output. They achieve dimensionality reduction by learning a bottleneck layer with a much smaller dimension than the input. The network consists of an encoder (which maps input to the bottleneck) and a decoder (which reconstructs the input from the bottleneck). By minimizing the reconstruction error (e.g., Mean Squared Error), the network learns a compressed representation that captures the most salient features of the data. This is a non-linear generalization of PCA.

This figure illustrates the complete architecture of an autoencoder, which is a neural network-based approach to dimensionality reduction. The input is a high-dimensional vector (e.g., a 784-pixel image flattened into a 784-dimensional vector). The encoder, a feedforward neural network with non-linear activations (like ReLU), maps this input to a much smaller "Code" (bottleneck) of dimension 2 to 32. This code is the compressed representation. The decoder, another neural network, attempts to reconstruct the original input from this code. The entire network is trained by minimizing the reconstruction loss (e.g., Mean Squared Error between the input and the output). The key insight is that the network must learn to compress the information into the bottleneck and then expand it back out. If the bottleneck is too small, the network is forced to learn the most salient features of the data. This makes autoencoders a powerful non-linear generalization of PCA. Unlike PCA, which only allows linear projections, autoencoders can learn complex, curved manifolds, making them more effective for data like images, audio, and text.

5.1 Undercomplete Autoencoders

An undercomplete autoencoder has a bottleneck dimension smaller than the input dimension. The objective is:

$$L=||X-f_{\theta }(g_{\phi }(X)){||}^2$$

With non-linear activations, it can learn complex manifolds, making it superior to PCA for data with non-linear structure.

The undercomplete autoencoder forces the network to learn a compressed representation by limiting the capacity of the bottleneck. This is similar to PCA, but with the added flexibility of non-linear transformations. For example, an undercomplete autoencoder trained on MNIST digits can learn a 2D code that separates the digit classes in a non-linear manner, whereas PCA would struggle. The reconstruction loss ensures that the code retains enough information to regenerate the original input. The network is trained using backpropagation and gradient descent, often with the Adam optimizer. Regularization techniques like dropout and weight decay are essential to prevent overfitting, especially when the bottleneck is not extremely small.

5.2 Denoising Autoencoders

A denoising autoencoder reconstructs the original clean input from a corrupted version (e.g., Gaussian noise). The loss is:

$$L=||X-f_{\theta }(g_{\phi }(X_{noisy})){||}^2$$

This forces the network to learn robust features.

Denoising autoencoders are particularly effective for image denoising, where the network learns to remove noise and reconstruct the original clean image. The corruption can be Gaussian noise, masking noise (setting random pixels to zero), or salt-and-pepper noise. By training on corrupted inputs, the network learns to ignore noise and capture the underlying data manifold. This makes the learned representation more robust and less sensitive to small perturbations. Denoising autoencoders have been used as a pre-training step for deep networks, especially in scenarios with limited labeled data, and they are also used in anomaly detection, where high reconstruction error indicates an outlier.

5.3 Variational Autoencoders (VAEs)

Variational Autoencoders map the input to a distribution over the latent space. The loss is the Evidence Lower Bound (ELBO):

$$L(\theta ,\phi ;x)=E_{q_{\phi }(z|x)}[\log p_{\theta }(x|z)]-\mathrm{KL}(q_{\phi }(z|x)||p(z))$$

This regularizes the latent space, making it continuous and generative.

VAEs are a cornerstone of generative modeling. By learning a continuous latent space, they allow smooth interpolation between data points. For example, in the case of face images, moving along a latent dimension can change facial expressions or age. The KL divergence term acts as a regularizer, preventing the latent space from becoming too sparse and encouraging it to be Gaussian-like. This enables the VAE to generate new samples by sampling from the prior distribution and decoding. VAEs have been used for image generation, text generation, and even drug molecule design. However, they can produce blurry images compared to GANs, as the reconstruction loss encourages averaging.

5.4 Other Autoencoder Variants

Contractive Autoencoders add a penalty on the Frobenius norm of the encoder's Jacobian. Sparse Autoencoders impose a sparsity constraint on the hidden representation. VQ-VAE learns a discrete latent space using a learned codebook.

These variants address specific limitations of vanilla autoencoders. Contractive autoencoders encourage the representation to be insensitive to small changes in the input, making them robust and capturing the manifold structure. Sparse autoencoders learn a sparse, overcomplete representation, which has been shown to learn biologically plausible features (e.g., Gabor filters). VQ-VAE discretizes the latent space, which is useful for tasks like text-to-speech and image generation, as it allows for a finite set of latent codes that can be modeled with autoregressive models like PixelCNN or Transformers. VQ-VAE and its variants (VQ-VAE-2) have been used in state-of-the-art models like DALL-E and Muse, demonstrating the power of discrete representations.

Applications

• Data compression (image denoising, super-resolution).
• Anomaly detection.
• Generative modeling.
• Feature extraction for downstream tasks.

Strengths and limitations

Strengths	Limitations
Non-linear and highly expressive.	Requires large amounts of data to train.
Can learn complex, hierarchical features.	Prone to overfitting without regularization.
VAEs provide a generative latent space.	Training is computationally expensive.

6. Other Classical Methods: LDA, MDS, and Isomap

Linear Discriminant Analysis (LDA) is a supervised method that focuses on class separability. Multidimensional Scaling (MDS) preserves pairwise distances. Isomap is an early, influential non-linear method that uses geodesic distances.

6.1 Linear Discriminant Analysis (LDA)

LDA maximizes the ratio of between-class variance to within-class variance:

$$J(w)=\frac{w^T{S}_{B}w}{w^T{S}_{W}w}$$

The solution is the generalized eigenvalue problem:

$$S_{B}w=\lambda S_{W}w$$

LDA reduces dimension to at most (number of classes − 1).

LDA is a powerful supervised linear reduction technique. It finds a projection that best separates the classes by considering both the scatter between classes and the scatter within classes. In practice, LDA is used in face recognition (Fisherfaces) and document classification. The within-class scatter matrix S_W is the covariance of the class-mean centered data, while S_B is the covariance of the class means. The projection vectors are the eigenvectors corresponding to the largest eigenvalues of S_W^{-1} S_B. LDA is optimal when the classes are Gaussian with equal covariance, but it can still be effective in many real-world scenarios. One limitation is that it requires labeled data and can only produce at most c-1 dimensions, where c is the number of classes.

6.2 Multidimensional Scaling (MDS)

Classical MDS takes a pairwise distance matrix and finds an embedding that preserves distances. It computes the Gram matrix:

$$B=-\frac{1}{2}J{D}^{2}J$$

and performs eigenvalue decomposition. If distances are Euclidean, it is equivalent to PCA.

MDS is widely used in psychometrics and marketing to create perceptual maps. For example, it can take the pairwise similarity ratings of different brands and plot them in 2D, revealing competitive positioning. The embedding coordinates are derived from the eigenvectors of B, scaled by the square root of the eigenvalues. MDS can handle any distance metric, making it flexible, but it is computationally expensive for large datasets (O(n³)). Non-metric MDS extends the idea to ordinal or categorical dissimilarities, preserving the rank order rather than the actual distances.

6.3 Isomap (Isometric Mapping)

Isomap extends MDS by replacing Euclidean distances with geodesic distances along the manifold. The algorithm: 1) Construct a neighborhood graph. 2) Compute shortest paths. 3) Apply classical MDS to the geodesic distance matrix. It is effective for continuous manifolds but is computationally expensive (O(n³)).

Isomap was one of the first methods to demonstrate the power of manifold learning. It is particularly effective for datasets like the "Swiss roll" where Euclidean distances would cut across the manifold, but geodesic distances follow the curvature. The neighborhood graph is typically constructed using k-nearest neighbors or an ε-ball. The shortest paths are computed using Dijkstra's algorithm. The resulting geodesic distance matrix is then embedded via MDS. Isomap is sensitive to the choice of neighborhood size; too small leads to disconnected components, too large approximates Euclidean distances. Despite its computational cost, it remains a valuable tool for understanding the intrinsic geometry of data.

Applications

• Face recognition (LDA).
• Perceptual and preference mapping (MDS).
• Visualization of high-dimensional manifolds (Isomap).

Strengths and limitations

Strengths	Limitations
LDA is optimal for classification.	LDA requires labeled data.
MDS works with any distance metric.	MDS and Isomap are O(N³) computationally.
Isomap discovers non-linear structure.	Isomap is sensitive to parameter choices (k).

7. Feature Selection: Filter, Wrapper and Embedded

Feature selection directly selects a subset of the original features, preserving interpretability and reducing overfitting. The three paradigms are Filter (statistical pre-processing), Wrapper (model-based search), and Embedded (built-in regularization).

This figure provides a high-level overview of the three main paradigms of feature selection, which is a distinct form of dimensionality reduction that preserves the original meaning of the features. The Filter method (blue) is the fastest and most scalable. It ranks features using statistical measures like correlation or mutual information, independent of any model. This is like a pre-screening step that removes irrelevant features before any modeling begins. The Wrapper method (orange) is more accurate but computationally expensive. It evaluates subsets of features by actually training a model and measuring its performance (e.g., accuracy). This is like a search problem, where features are added or removed iteratively to find the best performing subset. The Embedded method (green) finds a middle ground by performing feature selection during model training. For example, Lasso regression adds a penalty term that drives the coefficients of unimportant features to exactly zero, effectively selecting a subset. This is built into the model, making it more efficient than wrappers but often more accurate than filters. The choice among these depends on the dataset size, the importance of interpretability, and the computational budget.

7.1 Filter Methods

Filter methods rank features based on statistical properties. Common metrics include Pearson correlation, Chi-square test, and Mutual Information. They are computationally efficient and model-agnostic.

Filter methods are often used as a first step to reduce dimensionality from thousands to hundreds of features. For example, in text classification, mutual information is used to select the most informative words. The main advantage is speed; they can handle datasets with millions of features. However, they ignore feature interactions and the specific model's inductive bias. A common practice is to combine filter methods with wrapper or embedded methods for a two-stage selection process.

7.2 Wrapper Methods

Wrapper methods evaluate subsets by training a model and measuring performance. Forward selection, backward elimination, and Recursive Feature Elimination (RFE) are standard strategies. They find better subsets but are computationally intensive.

Wrapper methods are more accurate than filters because they consider the interaction between features and the model. RFE, for example, recursively removes the least important features based on model coefficients or feature importance. The number of features to retain is often chosen by cross-validation. Wrapper methods are feasible for moderate feature counts (e.g., up to a few thousand). For very high-dimensional data, they become prohibitively expensive. Hybrid approaches, like using a filter first to reduce to a manageable size, are common.

7.3 Embedded Methods

Embedded methods perform feature selection during training. Lasso (L1 regularization) drives coefficients to zero:

$$\underset{w}{min}||y-Xw{||}^2+\lambda ||w{||}_1$$

Decision trees and random forests inherently rank features via impurity decrease.

Embedded methods offer the best trade-off between speed and accuracy. Lasso is widely used in high-dimensional regression problems, such as genomic data analysis, where the number of features (genes) far exceeds the number of samples. The regularization path can be computed efficiently using coordinate descent. For tree-based models, feature importance is computed as the average reduction in impurity (Gini or entropy) across all trees. These importance scores are then used to select a subset of features. Embedded methods are model-specific, meaning the selected features are optimal for the specific model used.

Applications

• Text classification (selecting informative words).
• Genomics (identifying disease-related biomarkers).
• High-dimensional fraud detection.

Strengths and limitations

Strengths	Limitations
Preserves interpretability.	Wrapper methods are computationally expensive.
Reduces overfitting.	Filter methods ignore feature interactions.
Embedded methods balance speed and accuracy.	Lasso performs poorly with highly correlated features.

8. Independent Component Analysis (ICA) and Blind Source Separation

Independent Component Analysis (ICA) separates a multivariate signal into additive subcomponents that are statistically independent. It is the standard solution to the Cocktail Party Problem. ICA seeks non-Gaussianity (e.g., kurtosis or negentropy) to recover independent sources.

This figure illustrates the classic Cocktail Party Problem, which ICA solves. On the left, we have two recorded mixtures of two different speakers (e.g., two microphones at a party). Each mixture contains overlapping speech signals from both speakers. The "ICA Unmixing" block represents the algorithm that computes an unmixing matrix W such that the output sources are statistically independent. The results are shown on the right: Source 1 contains only the speech of Speaker 1, and Source 2 contains only the speech of Speaker 2. The key to ICA is that it does not require any knowledge of the mixing process or the original sources. It relies on the statistical property of independence: if the sources are independent, and the mixtures are linear combinations, then maximizing the non-Gaussianity of the outputs recovers the independent components. This is a powerful example of unsupervised dimensionality reduction, where the original 2-dimensional mixed signal is transformed into a 2-dimensional independent representation that is far more interpretable. ICA is also used to separate brain signals (EEG) from muscle artifacts and to find independent features in images.

8.1 Mathematical Foundations

The ICA model is:

$$X=AS$$

The goal is to find:

$$Y=WX$$

such that Y are independent. Maximizing non-Gaussianity (e.g., kurtosis or negentropy) recovers the sources. FastICA is a popular fixed-point algorithm.

The independence assumption is stronger than uncorrelatedness, which PCA captures. ICA seeks components that are as non-Gaussian as possible, as the central limit theorem states that mixtures of independent sources tend to be more Gaussian. The FastICA algorithm uses a fixed-point iteration to find the directions of maximum negentropy. ICA is often used as a preprocessing step for feature extraction, and its components can be used for classification or clustering. However, ICA is sensitive to noise and the order of the components is arbitrary (permutation ambiguity), and the signs are also ambiguous.

8.2 Applications

• Audio signal processing (speech separation).
• Biomedical signal analysis (EEG artifact removal).
• Image feature extraction.
• Financial time series analysis.

ICA has been successfully applied in many domains. In EEG, it separates brain signals from eye blink and muscle artifacts, allowing for cleaner analysis. In finance, ICA is used to identify independent risk factors that drive asset returns. In image processing, ICA can find independent features, such as edges or texture primitives, which are useful for image coding and classification. Despite its power, ICA requires careful preprocessing, including centering and whitening, and its performance degrades in the presence of additive noise. Recent advances include robust ICA and variants that handle non-linear mixtures.

Strengths and limitations

Strengths	Limitations
Powerful for blind source separation.	Assumes statistical independence.
Does not require labeled data.	Sensitive to noise and outliers.
Finds non-orthogonal basis.	Order of sources is arbitrary.

9. Manifold Learning II: LLE and Spectral Methods

Locally Linear Embedding (LLE) and Laplacian Eigenmaps are non-linear methods based on local linearity and graph Laplacians. LLE preserves local reconstruction weights, while Laplacian Eigenmaps uses the graph Laplacian to preserve local neighborhoods.

This figure shows the conceptual process of Locally Linear Embedding (LLE). On the left, we see a high-dimensional manifold shaped like a "Swiss roll" (a common benchmark for manifold learning). The points are colored to indicate their position along the manifold. LLE assumes that each point and its neighbors lie on a locally linear patch of the manifold. It computes reconstruction weights that express each point as a linear combination of its neighbors, and then finds a low-dimensional embedding that preserves these weights. The result, shown on the right, is the "unfolded" 2D embedding where the colors reveal the underlying structure. LLE is particularly effective for data with continuous, smooth variations, such as images of a rotating object or a person walking. It preserves the local distances and topology, making it a valuable tool for visualization and feature extraction. Unlike t-SNE, which is stochastic, LLE has a deterministic solution derived from eigenvalue decomposition, but it is sensitive to the choice of the neighborhood size k.

9.1 Locally Linear Embedding (LLE)

LLE has three steps: 1) Find k-nearest neighbors. 2) Compute reconstruction weights W minimizing:

$$\epsilon (W)=\underset{i}{\Sigma }||x_i-\underset{j}{\Sigma }{W}_{ij}{x}_j{||}^2$$

subject to $\underset{j}{\Sigma }{W}_{ij}=1$. 3) Find low-dimensional coordinates Y minimizing:

$$\Phi (Y)=\underset{i}{\Sigma }||y_i-\underset{j}{\Sigma }{W}_{ij}{y}_j{||}^2$$

The reconstruction weights are computed by solving a least-squares problem for each point. The constraint that the weights sum to 1 ensures that the solution is translation-invariant. The low-dimensional coordinates are obtained by computing the eigenvectors of the sparse matrix M = (I-W)^T(I-W), corresponding to the smallest non-zero eigenvalues. LLE is computationally efficient for datasets up to a few thousand points, but it scales poorly to very large datasets due to the eigenvalue decomposition. The choice of k is critical; a common heuristic is to set k to be greater than the intrinsic dimension of the data.

9.2 Laplacian Eigenmaps

Laplacian Eigenmaps constructs a graph with weights:

$$W_{ij}=\exp (-||x_i-x_j{||}^2/2{\sigma }^2)$$

It solves the generalized eigenvalue problem:

$$Lf=\lambda Df$$

where L is the Laplacian. The eigenvectors with the smallest non-zero eigenvalues give the embedding.

Laplacian Eigenmaps is based on spectral graph theory. The graph Laplacian L = D - W (where D is the degree matrix) represents the local neighborhood structure. The eigenvectors of the Laplacian provide a low-dimensional embedding that preserves local distances. The parameter σ controls the scale of the heat kernel; a small σ makes the graph highly localized. Laplacian Eigenmaps is computationally similar to LLE but can handle larger graphs due to sparse eigensolvers. It is often used as a preprocessing step for clustering (spectral clustering) and for semi-supervised learning, where the graph structure is used to propagate labels.

9.3 Applications

• Visualization of complex scientific data.
• Motion capture data analysis.
• Image manifold unfolding.
• Spectral clustering.

Both LLE and Laplacian Eigenmaps are powerful tools for exploratory analysis of high-dimensional data. In motion capture, LLE can be used to understand the low-dimensional manifold of human poses, enabling motion synthesis and recognition. In image processing, these methods can unfold manifolds of object poses, such as a rotating face or a walking person. Spectral clustering, which uses the eigenvectors of the Laplacian for clustering, is a direct application of Laplacian Eigenmaps and is widely used in community detection and image segmentation.

Strengths and limitations

Strengths	Limitations
LLE preserves local structure excellently.	Sensitive to the choice of k.
Laplacian Eigenmaps handles non-linear manifolds well.	Computationally expensive for large datasets.
Strong theoretical foundation in spectral graph theory.	Out-of-sample extension is not trivial.

10. Non-Negative Matrix Factorization (NMF) and Dictionary Learning

Non-Negative Matrix Factorization (NMF) factorizes a non-negative matrix V into non-negative matrices W (basis) and H (coefficients): $V\approx WH$. This parts-based representation is highly interpretable and is standard for topic modeling and facial feature extraction.

This figure shows the mathematical factorization that NMF performs. The data matrix V (size n × p, where n is samples and p is features) is decomposed into W (n × k, the basis or dictionary) and H (k × p, the coefficients). The key constraint is that all three matrices must be non-negative (W ≥ 0, H ≥ 0). This non-negativity leads to a parts-based representation because there are no subtractions; everything is additive. For example, if V represents a set of face images, W might learn "basis faces" like eyes, noses, and mouths, and H would represent how these parts are combined to form each face. This additive nature makes NMF highly interpretable, as the components often correspond to semantically meaningful parts. For text data, W can represent topics (e.g., "sports," "politics," "technology"), and H represents the topic distribution for each document. NMF is a powerful tool for dimensionality reduction because the rank k is typically much smaller than p, reducing the number of features to a manageable, interpretable set.

10.1 Mathematical Formulation

NMF solves:

$$\underset{W,H}{min}||V-WH{||}_F^2$$

subject to $W,H\ge 0$. Multiplicative update rules:

$$H\leftarrow H\odot \frac{W^{T}V}{W^{T}WH}$$

The parameter k dictates the reduced dimension.

The multiplicative updates are derived from the gradient of the Frobenius norm and are guaranteed to converge to a local minimum. They are simple to implement and maintain non-negativity. However, NMF is sensitive to initialization and can converge to different local minima. Common initialization methods include random initialization, SVD-based initialization, and non-negative double singular value decomposition (NNDSVD). The choice of k is often guided by the desired level of compression or by cross-validation for downstream tasks. NMF has been extended to handle missing data and to incorporate sparsity constraints, making it a versatile tool.

10.2 Dictionary Learning and Sparse Coding

Sparse Dictionary Learning imposes sparsity on H:

$$\underset{W,H}{min}||V-WH{||}_F^2+\lambda ||H{||}_1$$

Algorithms like OMP (greedy) or Lasso solve sparse coding; KSVD updates the dictionary.

Dictionary learning goes beyond NMF by allowing an overcomplete basis (k > p) and forcing sparse coefficients. This is powerful for signal processing tasks like image denoising, inpainting, and compression. The sparse representation captures the essential structure with few non-zero coefficients, making it robust and interpretable. The alternating optimization (sparse coding and dictionary update) is computationally efficient and can be applied to large datasets. KSVD is a popular dictionary learning algorithm that updates each dictionary atom and the corresponding coefficients sequentially. Sparse coding is also used in feature learning for deep networks, where sparse representations can improve classification performance.

Applications

• Topic modeling in text documents.
• Image feature extraction.
• Hyperspectral unmixing.
• Recommendation systems.

Strengths and limitations

Strengths	Limitations
Highly interpretable parts-based representation.	Requires non-negative input data.
Works well for text and image data.	Non-convex optimization can get stuck in local minima.
Sparse variants are robust and efficient.	Choosing the right rank k is non-trivial.

11. Random Projection and the Johnson-Lindenstrauss Lemma

Random Projection is a computationally cheap technique grounded in the Johnson-Lindenstrauss Lemma. It states that high-dimensional data can be projected into $O(\log n/{\epsilon }^2)$ dimensions while approximately preserving pairwise distances.

This figure illustrates the concept of Random Projection, one of the simplest and fastest dimensionality reduction techniques. On the left, we have high-dimensional data points (blue circles) in a space of dimension p. The arrow represents the application of a random matrix R of size k × p, where k is the target lower dimension. The matrix R can be Gaussian (entries from N(0,1)) or sparse (entries from {-1, 0, 1} with controlled sparsity). The result on the right is the data mapped to k dimensions (green circles). The Johnson-Lindenstrauss lemma guarantees that with high probability, the pairwise distances between points are preserved within a factor of (1 ± ε), as long as k is at least $O(\log n/{\epsilon }^2)$. This is a powerful result because k is independent of the original dimension p. Random projection is extremely fast because it only involves matrix multiplication, and for sparse matrices, this can be even faster. It is used extensively in streaming data, approximate nearest neighbor search, and as a preprocessing step for more expensive methods like PCA.

11.1 The Johnson-Lindenstrauss Lemma

The JL lemma states that for any set of n points, there exists a map to $k=O(\log n/{\epsilon }^2)$ dimensions such that:

$$(1-\epsilon )||u-v{||}^2\le ||f(u)-f(v){||}^2\le (1+\epsilon )||u-v{||}^2$$

The target dimension is independent of the original dimension.

The lemma's proof uses the fact that random projections are almost isometries. The constant factor in the O(log n) term is small, typically around 4. For example, to preserve distances within 10% (ε=0.1) for n=10,000 points, k≈4600, which is far smaller than the original dimension if p is large. This makes random projection a practical tool for dimensionality reduction. The lemma is also the theoretical basis for compressive sensing, where signals are recovered from fewer measurements. In practice, random projection is often used with the Gaussian or Achlioptas sparse matrix, and the results are surprisingly good for many applications.

11.2 Practical Implementation

Generate a random matrix R (size p × k) with Gaussian or sparse entries, and compute:

$$Y=XR$$

This is extremely fast and memory-efficient.

In practice, the random matrix can be generated on the fly or stored once. For sparse matrices, the multiplication can be done in O(n * p * s) where s is the sparsity (e.g., s=3 for the Achlioptas matrix). This makes random projection scalable to datasets with millions of features. Libraries like scikit-learn provide efficient implementations. Random projection is often used as a preprocessing step for k-means clustering or nearest neighbor search, where the reduced dimension significantly speeds up computation with minimal loss in accuracy. It is also a key component in the "randomized SVD" algorithm, which approximates the SVD of large matrices.

Applications

• Accelerating distance-based algorithms (K-means, KNN).
• Fast Fourier transform and signal processing.
• Compressive sensing.
• Preprocessing for extremely high-dimensional datasets.

Strengths and limitations

Strengths	Limitations
Extremely fast and memory-efficient.	Random nature means results are non-deterministic.
Does not depend on the data distribution.	The projected space is not interpretable.
Guarantees distance preservation (JL lemma).	Can perform poorly if structure is not Euclidean.

12. Additional Specialized Methods

Factor Analysis (FA) explains covariance via latent factors. Canonical Correlation Analysis (CCA) finds relationships between two sets of variables. TriMap is a newer non-linear method that balances local and global structure better than t-SNE.

12.1 Factor Analysis (FA)

FA is a linear latent variable model:

$$X=\mu +Lf+\epsilon$$

It explicitly models unique variance (noise). The covariance matrix is:

$$\Sigma =L{L}^T+\Psi$$

It is widely used in psychometrics.

Factor analysis is a cornerstone of psychometrics and social sciences. It aims to explain the correlations among observed variables by uncovering a smaller number of latent factors. For example, in intelligence testing, a single "g" factor (general intelligence) is often used to explain the correlations across different cognitive tasks. The factor loadings L indicate the strength of each variable's relationship to the latent factors. FA is often confused with PCA, but they differ: PCA is a data transformation that captures variance, while FA is a statistical model that explains covariance. FA also assumes that the unique variances (Ψ) are uncorrelated and often uses maximum likelihood estimation. In practice, FA is used in survey analysis, market research, and any domain where latent constructs are of interest.

12.2 Canonical Correlation Analysis (CCA)

CCA finds vectors a and b that maximize the correlation between $Xa$ and $Yb$. This is solved as a generalized eigenvalue problem. Deep CCA extends it with neural networks.

CCA is a powerful technique for multi-modal data analysis. For instance, in neuroscience, CCA can find the linear combination of brain activity (fMRI) that correlates best with behavioral measures. In computer vision, CCA can align image features with text descriptions, enabling cross-modal retrieval. Deep CCA (DCCA) uses deep neural networks to learn non-linear transformations for each view, significantly improving the correlation and representation learning. CCA has also been used in recommendation systems, where user and item representations are learned to maximize correlation. The number of canonical variates is at most min(p, q), providing a natural dimensionality reduction for each view.

12.3 UMAP variants and TriMap

TriMap minimizes a triplet-based cost function that balances local and global structure:

$$\max (0,||y_i-y_j{||}^2-||y_i-y_k{||}^2+\mathrm{margin})$$

It is computationally efficient and produces interpretable embeddings.

TriMap is a recent addition to the manifold learning arsenal, addressing some of the shortcomings of t-SNE and UMAP. It uses triplets (anchor, positive, negative) to enforce that the anchor is closer to the positive than to the negative in the embedding. By weighting the triplets appropriately, TriMap can preserve both local and global structure better than t-SNE. It is particularly effective for large datasets because it uses a stochastic optimization approach similar to UMAP. TriMap has been shown to produce embeddings that are more interpretable and stable, making it a promising alternative for visualization and exploratory analysis.

Applications

• Psychometrics and survey analysis (FA).
• Multimodal data fusion (CCA).
• Visualization of very large datasets (TriMap).

Strengths and limitations

Strengths	Limitations
Specialized for specific tasks.	Limited applicability beyond their niche.
CCA is excellent for multi-modal alignment.	TriMap is newer and has less community adoption.
TriMap balances local/global structure better.	Methods are often computationally heavier.

13. Choosing the Right Method and Evaluation Metrics

The choice of a dimensionality reduction method depends on the data, goals, and constraints. This section provides a practical guide and evaluation metrics.

13.1 Decision Tree for Method Selection

1. Is interpretability critical? Yes → Feature selection or NMF. No → Proceed.
2. Is the data linear or non-linear? Linear → PCA, SVD, LDA. Non-linear → t-SNE, UMAP, LLE.
3. What is the dataset size? Large (>50k) → UMAP, Random Projection. Small → Any method.
4. Is the data labeled? Labeled → LDA, supervised autoencoders. Unlabeled → PCA, t-SNE, UMAP.
5. Are there outliers? Yes → Robust PCA, denoising autoencoders.

This decision tree is a starting point. In practice, it is recommended to try multiple methods and evaluate their performance on a downstream task or through visualization. For example, if you are clustering, you can measure cluster purity after reduction. If you are classifying, you can measure accuracy. The choice of method is often iterative, and the best approach may involve a combination of techniques (e.g., PCA followed by t-SNE for visualization).

13.2 Evaluation Metrics

Reconstruction Error: MSE between original and reconstructed data.
Explained Variance: For PCA, the fraction of variance retained.
Trustworthiness & Continuity: Measure local neighborhood preservation.
Downstream Task Performance: Accuracy or clustering purity after reduction.
Computational Efficiency: Runtime and memory usage.

Evaluation is crucial to ensure that the reduction is effective. For linear methods, reconstruction error and explained variance are straightforward. For non-linear methods, trustworthiness and continuity are more appropriate. These metrics compare the neighborhoods in the original space to those in the reduced space. A high trustworthiness score indicates that points that are close in the reduced space are also close in the original space, and vice versa for continuity. Ultimately, the best evaluation is the performance on the intended task. If the reduced representation improves or maintains performance while reducing computational cost, the reduction is successful.

Key takeaways

• Statistical Summaries collapse data to single values (e.g., averages allow easy comparison of students).
• Linear Matrix Factorization (PCA, SVD, ICA) finds projections based on variance or independence.
• Pooling is a non-parametric reduction essential for CNNs.
• Manifold Learning (t-SNE, UMAP, LLE) preserves local neighborhoods for non-linear data.
• Autoencoders provide expressive, non-linear compression and generative capabilities.
• Feature Selection preserves interpretability.
• NMF provides parts-based representations.
• Random Projection leverages the JL lemma for fast, distance-preserving reduction.
• The choice of method depends on data size, linearity, labels, and the ultimate goal.

Resources

The sources below are seminal papers and textbooks that define the field of dimensionality reduction.

• [1] Principal Component Analysis — Pearson, K. (1901). Philosophical Magazine. doi.org
• [2] The Singular Value Decomposition — Golub & Van Loan (1983). Matrix Computations. jstor.org
• [3] Visualizing Data using t-SNE — van der Maaten & Hinton (2008). JMLR. jmlr.org
• [4] UMAP: Uniform Manifold Approximation and Projection — McInnes, Healy, Melville (2018). arxiv.org
• [5] Auto-Encoding Variational Bayes — Kingma & Welling (2014). ICLR. arxiv.org
• [6] A Global Geometric Framework for Nonlinear Dimensionality Reduction — Tenenbaum, de Silva, Langford (2000). Science. doi.org
• [7] Linear Discriminant Analysis — Fisher, R. A. (1936). Annals of Eugenics. doi.org
• [8] Learning the Parts of Objects by Non-Negative Matrix Factorization — Lee & Seung (1999). Nature. doi.org
• [9] Extensions of Lipschitz mappings into a Hilbert space — Johnson & Lindenstrauss (1984). Contemporary Mathematics. doi.org
• [10] Locally Linear Embedding — Roweis & Saul (2000). Science. doi.org
• [11] Independent Component Analysis: Algorithms and Applications — Hyvärinen & Oja (2000). Neural Networks. doi.org
• [12] Relations Between Two Sets of Variates — Hotelling (1936). Biometrika. doi.org
• [13] TriMap: Large-scale Dimensionality Reduction Using Triplets — Amid & Warmuth (2022). JMLR. jmlr.org