7. Feature Selection: Filter, Wrapper and Embedded
Feature selection directly selects a subset of the original features, preserving interpretability and reducing overfitting. The three paradigms are Filter (statistical pre-processing), Wrapper (model-based search), and Embedded (built-in regularization).
This figure provides a high-level overview of the three main paradigms of feature selection, which is a distinct form of dimensionality reduction that preserves the original meaning of the features. The Filter method (blue) is the fastest and most scalable. It ranks features using statistical measures like correlation or mutual information, independent of any model. This is like a pre-screening step that removes irrelevant features before any modeling begins. The Wrapper method (orange) is more accurate but computationally expensive. It evaluates subsets of features by actually training a model and measuring its performance (e.g., accuracy). This is like a search problem, where features are added or removed iteratively to find the best performing subset. The Embedded method (green) finds a middle ground by performing feature selection during model training. For example, Lasso regression adds a penalty term that drives the coefficients of unimportant features to exactly zero, effectively selecting a subset. This is built into the model, making it more efficient than wrappers but often more accurate than filters. The choice among these depends on the dataset size, the importance of interpretability, and the computational budget.
7.1 Filter Methods
Filter methods rank features based on statistical properties. Common metrics include Pearson correlation, Chi-square test, and Mutual Information. They are computationally efficient and model-agnostic.
Filter methods are often used as a first step to reduce dimensionality from thousands to hundreds of features. For example, in text classification, mutual information is used to select the most informative words. The main advantage is speed; they can handle datasets with millions of features. However, they ignore feature interactions and the specific model's inductive bias. A common practice is to combine filter methods with wrapper or embedded methods for a two-stage selection process.
7.2 Wrapper Methods
Wrapper methods evaluate subsets by training a model and measuring performance. Forward selection, backward elimination, and Recursive Feature Elimination (RFE) are standard strategies. They find better subsets but are computationally intensive.
Wrapper methods are more accurate than filters because they consider the interaction between features and the model. RFE, for example, recursively removes the least important features based on model coefficients or feature importance. The number of features to retain is often chosen by cross-validation. Wrapper methods are feasible for moderate feature counts (e.g., up to a few thousand). For very high-dimensional data, they become prohibitively expensive. Hybrid approaches, like using a filter first to reduce to a manageable size, are common.
7.3 Embedded Methods
Embedded methods perform feature selection during training. Lasso (L1 regularization) drives coefficients to zero:
Decision trees and random forests inherently rank features via impurity decrease.
Embedded methods offer the best trade-off between speed and accuracy. Lasso is widely used in high-dimensional regression problems, such as genomic data analysis, where the number of features (genes) far exceeds the number of samples. The regularization path can be computed efficiently using coordinate descent. For tree-based models, feature importance is computed as the average reduction in impurity (Gini or entropy) across all trees. These importance scores are then used to select a subset of features. Embedded methods are model-specific, meaning the selected features are optimal for the specific model used.
Applications
- Text classification (selecting informative words).
- Genomics (identifying disease-related biomarkers).
- High-dimensional fraud detection.
Strengths and limitations
| Strengths | Limitations |
|---|---|
| Preserves interpretability. | Wrapper methods are computationally expensive. |
| Reduces overfitting. | Filter methods ignore feature interactions. |
| Embedded methods balance speed and accuracy. | Lasso performs poorly with highly correlated features. |