Dimension Reduction Taxonomy - 1. Statistical Summaries: Mean, Median and Variance

1. Statistical Summaries: Mean, Median and Variance

The most fundamental form of dimensionality reduction is taking the simple average (arithmetic mean). When faced with a vector of numbers (e.g., a student's marks in different subjects), we collapse it to a single scalar. The median serves as a robust alternative. These methods reduce dimension from n to 1. They are parameter-free, interpretable, and computationally optimal.

For example, consider Student A with marks [90, 85, 88] in Mathematics, Physics, and Chemistry, and Student B with [70, 75, 80]. By averaging, Student A's profile reduces to 87.7, and Student B's to 75.0. This dramatic reduction from 3 dimensions to 1 allows for an immediate and intuitive comparison: Student A is outperforming Student B overall. The diagram in Fig 2 visualizes this exact process.

This figure illustrates the power of the arithmetic mean as a dimensionality reduction tool. Student A's scores are displayed as bars of varying heights: 90 in Math, 85 in Physics, and 88 in Chemistry. Each of these is a separate dimension. The arrow points to a single red circle containing the average, 87.7. Student B's scores (70, 75, 80) are similarly collapsed to a green circle with the value 75.0. By reducing each student's 3-dimensional profile to a 1-dimensional scalar, we can instantly compare the two students: A is performing better overall. This example demonstrates the fundamental trade-off of all reduction methods: we lose subject-specific information (e.g., Student B might be stronger in Chemistry than A, despite a lower overall average), but we gain a compact, comparable summary that is useful for ranking and decision-making.

1.1 Arithmetic Mean (Simple Average)

The arithmetic mean is defined as:

It minimizes the sum of squared errors. For the example in Fig 2, Student A's average is 87.7, and Student B's is 75.0. These single numbers capture the overall performance level of each student, allowing a direct comparison. In feature engineering, averages are used to normalize data (e.g., subtracting the mean) and to create aggregated features.

Mathematical properties: The mean is the minimum-variance unbiased estimator for the location parameter of a Gaussian distribution. It satisfies the linearity property: for constants a and b,

The mean is also the solution to the least-squares problem:

which is why it is the natural choice for minimizing reconstruction error in many statistical models. The mean is sensitive to every data point, which is both a strength (it uses all information) and a weakness (outliers can skew it). In practice, the mean is computed in O(n) time and is a fundamental building block for more complex algorithms like PCA, where data is centered by subtracting the mean.

1.2 Median

The median is the middle value when observations are ordered. For an odd number of observations, it is the central element; for an even number, it is the average of the two central elements. It minimizes the sum of absolute deviations. The median is robust; an outlier does not shift it. This makes the median invaluable in income data, sensor readings prone to glitches, and any domain where outliers must not distort the summary. The median absolute deviation (MAD), defined as

is a robust scale estimator often used in feature filtering.

The median has a breakdown point of 50%, meaning that up to half of the data can be arbitrarily corrupted without changing the median. This property makes it the preferred summary for skewed distributions, such as household income or house prices. In high-dimensional settings, computing the median is more expensive than the mean, requiring O(n log n) time or O(n) with selection algorithms. However, for many robust applications, the median is indispensable, especially when combined with MAD for outlier detection and feature scaling.

1.3 Variance and Standard Deviation

While not a reduction in dimension on its own, variance measures the spread of data and is the objective function for Principal Component Analysis. Variance is defined as:

Feature selection often relies on the low variance filter: features with variance near zero are considered constant and are removed. This is a straightforward reduction step, effectively reducing the number of features by eliminating those that carry little information.

The standard deviation σ is the square root of the variance and has the same units as the original data, making it more interpretable. In practice, the variance is computed using the two-pass formula for numerical stability. The coefficient of variation (CV = σ/μ) is a normalized measure of dispersion useful for comparing features with different scales. Variance is the cornerstone of PCA, where we seek projections that maximize the retained variance, effectively reducing dimensionality while preserving the most informative directions of the data.

Applications

• Baseline forecasting (e.g., average sales per day).
• Robust outlier filtering (median absolute deviation).
• Feature selection via low-variance filtering.

Strengths and limitations