Dimension Reduction Taxonomy - 12. Additional Specialized Methods

12. Additional Specialized Methods

Factor Analysis (FA) explains covariance via latent factors. Canonical Correlation Analysis (CCA) finds relationships between two sets of variables. TriMap is a newer non-linear method that balances local and global structure better than t-SNE.

12.1 Factor Analysis (FA)

FA is a linear latent variable model:

It explicitly models unique variance (noise). The covariance matrix is:

It is widely used in psychometrics.

Factor analysis is a cornerstone of psychometrics and social sciences. It aims to explain the correlations among observed variables by uncovering a smaller number of latent factors. For example, in intelligence testing, a single "g" factor (general intelligence) is often used to explain the correlations across different cognitive tasks. The factor loadings L indicate the strength of each variable's relationship to the latent factors. FA is often confused with PCA, but they differ: PCA is a data transformation that captures variance, while FA is a statistical model that explains covariance. FA also assumes that the unique variances (Ψ) are uncorrelated and often uses maximum likelihood estimation. In practice, FA is used in survey analysis, market research, and any domain where latent constructs are of interest.

12.2 Canonical Correlation Analysis (CCA)

CCA finds vectors a and b that maximize the correlation between $Xa$ and $Yb$. This is solved as a generalized eigenvalue problem. Deep CCA extends it with neural networks.

CCA is a powerful technique for multi-modal data analysis. For instance, in neuroscience, CCA can find the linear combination of brain activity (fMRI) that correlates best with behavioral measures. In computer vision, CCA can align image features with text descriptions, enabling cross-modal retrieval. Deep CCA (DCCA) uses deep neural networks to learn non-linear transformations for each view, significantly improving the correlation and representation learning. CCA has also been used in recommendation systems, where user and item representations are learned to maximize correlation. The number of canonical variates is at most min(p, q), providing a natural dimensionality reduction for each view.

12.3 UMAP variants and TriMap

TriMap minimizes a triplet-based cost function that balances local and global structure:

It is computationally efficient and produces interpretable embeddings.

TriMap is a recent addition to the manifold learning arsenal, addressing some of the shortcomings of t-SNE and UMAP. It uses triplets (anchor, positive, negative) to enforce that the anchor is closer to the positive than to the negative in the embedding. By weighting the triplets appropriately, TriMap can preserve both local and global structure better than t-SNE. It is particularly effective for large datasets because it uses a stochastic optimization approach similar to UMAP. TriMap has been shown to produce embeddings that are more interpretable and stable, making it a promising alternative for visualization and exploratory analysis.

Applications

• Psychometrics and survey analysis (FA).
• Multimodal data fusion (CCA).
• Visualization of very large datasets (TriMap).

Strengths and limitations