Dimension Reduction Taxonomy - 5. Deep Learning for Reduction: Autoencoders

5. Deep Learning for Reduction: Autoencoders

Autoencoders are neural networks trained to replicate their input at the output. They achieve dimensionality reduction by learning a bottleneck layer with a much smaller dimension than the input. The network consists of an encoder (which maps input to the bottleneck) and a decoder (which reconstructs the input from the bottleneck). By minimizing the reconstruction error (e.g., Mean Squared Error), the network learns a compressed representation that captures the most salient features of the data. This is a non-linear generalization of PCA.

This figure illustrates the complete architecture of an autoencoder, which is a neural network-based approach to dimensionality reduction. The input is a high-dimensional vector (e.g., a 784-pixel image flattened into a 784-dimensional vector). The encoder, a feedforward neural network with non-linear activations (like ReLU), maps this input to a much smaller "Code" (bottleneck) of dimension 2 to 32. This code is the compressed representation. The decoder, another neural network, attempts to reconstruct the original input from this code. The entire network is trained by minimizing the reconstruction loss (e.g., Mean Squared Error between the input and the output). The key insight is that the network must learn to compress the information into the bottleneck and then expand it back out. If the bottleneck is too small, the network is forced to learn the most salient features of the data. This makes autoencoders a powerful non-linear generalization of PCA. Unlike PCA, which only allows linear projections, autoencoders can learn complex, curved manifolds, making them more effective for data like images, audio, and text.

5.1 Undercomplete Autoencoders

An undercomplete autoencoder has a bottleneck dimension smaller than the input dimension. The objective is:

With non-linear activations, it can learn complex manifolds, making it superior to PCA for data with non-linear structure.

The undercomplete autoencoder forces the network to learn a compressed representation by limiting the capacity of the bottleneck. This is similar to PCA, but with the added flexibility of non-linear transformations. For example, an undercomplete autoencoder trained on MNIST digits can learn a 2D code that separates the digit classes in a non-linear manner, whereas PCA would struggle. The reconstruction loss ensures that the code retains enough information to regenerate the original input. The network is trained using backpropagation and gradient descent, often with the Adam optimizer. Regularization techniques like dropout and weight decay are essential to prevent overfitting, especially when the bottleneck is not extremely small.

5.2 Denoising Autoencoders

A denoising autoencoder reconstructs the original clean input from a corrupted version (e.g., Gaussian noise). The loss is:

This forces the network to learn robust features.

Denoising autoencoders are particularly effective for image denoising, where the network learns to remove noise and reconstruct the original clean image. The corruption can be Gaussian noise, masking noise (setting random pixels to zero), or salt-and-pepper noise. By training on corrupted inputs, the network learns to ignore noise and capture the underlying data manifold. This makes the learned representation more robust and less sensitive to small perturbations. Denoising autoencoders have been used as a pre-training step for deep networks, especially in scenarios with limited labeled data, and they are also used in anomaly detection, where high reconstruction error indicates an outlier.

5.3 Variational Autoencoders (VAEs)

Variational Autoencoders map the input to a distribution over the latent space. The loss is the Evidence Lower Bound (ELBO):

This regularizes the latent space, making it continuous and generative.

VAEs are a cornerstone of generative modeling. By learning a continuous latent space, they allow smooth interpolation between data points. For example, in the case of face images, moving along a latent dimension can change facial expressions or age. The KL divergence term acts as a regularizer, preventing the latent space from becoming too sparse and encouraging it to be Gaussian-like. This enables the VAE to generate new samples by sampling from the prior distribution and decoding. VAEs have been used for image generation, text generation, and even drug molecule design. However, they can produce blurry images compared to GANs, as the reconstruction loss encourages averaging.

5.4 Other Autoencoder Variants

Contractive Autoencoders add a penalty on the Frobenius norm of the encoder's Jacobian. Sparse Autoencoders impose a sparsity constraint on the hidden representation. VQ-VAE learns a discrete latent space using a learned codebook.

These variants address specific limitations of vanilla autoencoders. Contractive autoencoders encourage the representation to be insensitive to small changes in the input, making them robust and capturing the manifold structure. Sparse autoencoders learn a sparse, overcomplete representation, which has been shown to learn biologically plausible features (e.g., Gabor filters). VQ-VAE discretizes the latent space, which is useful for tasks like text-to-speech and image generation, as it allows for a finite set of latent codes that can be modeled with autoregressive models like PixelCNN or Transformers. VQ-VAE and its variants (VQ-VAE-2) have been used in state-of-the-art models like DALL-E and Muse, demonstrating the power of discrete representations.

Applications

• Data compression (image denoising, super-resolution).
• Anomaly detection.
• Generative modeling.
• Feature extraction for downstream tasks.

Strengths and limitations