Dimension Reduction Taxonomy - 13. Choosing the Right Method and Evaluation

13. Choosing the Right Method and Evaluation Metrics

The choice of a dimensionality reduction method depends on the data, goals, and constraints. This section provides a practical guide and evaluation metrics.

13.1 Decision Tree for Method Selection

1. Is interpretability critical? Yes → Feature selection or NMF. No → Proceed.
2. Is the data linear or non-linear? Linear → PCA, SVD, LDA. Non-linear → t-SNE, UMAP, LLE.
3. What is the dataset size? Large (>50k) → UMAP, Random Projection. Small → Any method.
4. Is the data labeled? Labeled → LDA, supervised autoencoders. Unlabeled → PCA, t-SNE, UMAP.
5. Are there outliers? Yes → Robust PCA, denoising autoencoders.

This decision tree is a starting point. In practice, it is recommended to try multiple methods and evaluate their performance on a downstream task or through visualization. For example, if you are clustering, you can measure cluster purity after reduction. If you are classifying, you can measure accuracy. The choice of method is often iterative, and the best approach may involve a combination of techniques (e.g., PCA followed by t-SNE for visualization).

13.2 Evaluation Metrics

Reconstruction Error: MSE between original and reconstructed data.
Explained Variance: For PCA, the fraction of variance retained.
Trustworthiness & Continuity: Measure local neighborhood preservation.
Downstream Task Performance: Accuracy or clustering purity after reduction.
Computational Efficiency: Runtime and memory usage.

Evaluation is crucial to ensure that the reduction is effective. For linear methods, reconstruction error and explained variance are straightforward. For non-linear methods, trustworthiness and continuity are more appropriate. These metrics compare the neighborhoods in the original space to those in the reduced space. A high trustworthiness score indicates that points that are close in the reduced space are also close in the original space, and vice versa for continuity. Ultimately, the best evaluation is the performance on the intended task. If the reduced representation improves or maintains performance while reducing computational cost, the reduction is successful.

Key takeaways

• Statistical Summaries collapse data to single values (e.g., averages allow easy comparison of students).
• Linear Matrix Factorization (PCA, SVD, ICA) finds projections based on variance or independence.
• Pooling is a non-parametric reduction essential for CNNs.
• Manifold Learning (t-SNE, UMAP, LLE) preserves local neighborhoods for non-linear data.
• Autoencoders provide expressive, non-linear compression and generative capabilities.
• Feature Selection preserves interpretability.
• NMF provides parts-based representations.
• Random Projection leverages the JL lemma for fast, distance-preserving reduction.
• The choice of method depends on data size, linearity, labels, and the ultimate goal.

Resources

The sources below are seminal papers and textbooks that define the field of dimensionality reduction.

• [1] Principal Component Analysis — Pearson, K. (1901). Philosophical Magazine. doi.org
• [2] The Singular Value Decomposition — Golub & Van Loan (1983). Matrix Computations. jstor.org
• [3] Visualizing Data using t-SNE — van der Maaten & Hinton (2008). JMLR. jmlr.org
• [4] UMAP: Uniform Manifold Approximation and Projection — McInnes, Healy, Melville (2018). arxiv.org
• [5] Auto-Encoding Variational Bayes — Kingma & Welling (2014). ICLR. arxiv.org
• [6] A Global Geometric Framework for Nonlinear Dimensionality Reduction — Tenenbaum, de Silva, Langford (2000). Science. doi.org
• [7] Linear Discriminant Analysis — Fisher, R. A. (1936). Annals of Eugenics. doi.org
• [8] Learning the Parts of Objects by Non-Negative Matrix Factorization — Lee & Seung (1999). Nature. doi.org
• [9] Extensions of Lipschitz mappings into a Hilbert space — Johnson & Lindenstrauss (1984). Contemporary Mathematics. doi.org
• [10] Locally Linear Embedding — Roweis & Saul (2000). Science. doi.org
• [11] Independent Component Analysis: Algorithms and Applications — Hyvärinen & Oja (2000). Neural Networks. doi.org
• [12] Relations Between Two Sets of Variates — Hotelling (1936). Biometrika. doi.org
• [13] TriMap: Large-scale Dimensionality Reduction Using Triplets — Amid & Warmuth (2022). JMLR. jmlr.org