5. Regularization Losses
Regularization losses are not used for fitting the data directly, but rather as penalty terms that encourage certain properties in the model, such as sparsity, smoothness, or small weights. They are often added to the primary loss function to prevent overfitting and improve generalization. The most common regularization losses are L1 Regularization (Lasso), L2 Regularization (Ridge), and Elastic Net, which combine both.
This figure compares the three main regularization penalties. The L1 penalty (blue) is piecewise linear and grows linearly with the absolute value of the weight. This encourages sparsity because the gradient is constant, and the optimal solution often occurs at zero. The L2 penalty (orange) is quadratic and grows with the square of the weight. This encourages small weights but does not force them to zero. The Elastic Net (green) is a combination of L1 and L2, with a parameter α controlling the balance between the two. Elastic Net encourages both sparsity and small weights.
5.1 L1 Regularization (Lasso)
L1 Regularization (also known as Lasso) adds the sum of the absolute values of the weights to the loss:
where λ is the regularization strength. L1 regularization encourages sparsity in the weights, making it useful for feature selection.
Mathematical properties: L1 regularization is a convex penalty that promotes sparsity in the model parameters. This is particularly useful in high-dimensional settings where many features are irrelevant. The L1 penalty forces some weights to exactly zero, effectively performing feature selection.
Real-World Applications: Feature selection in genomics (identifying the most relevant genes), software defect prediction (shrinking irrelevant coefficients to zero), sparse signal recovery, and diabetes dataset analysis (Lasso path visualization).
5.2 L2 Regularization (Ridge)
L2 Regularization (also known as Ridge) adds the sum of the squares of the weights to the loss:
L2 regularization encourages small weights but does not force them to zero. It is differentiable and has a gradient that is proportional to the weight.
Mathematical properties: L2 regularization is the most common regularization technique in deep learning. It encourages the model to keep the weights small, which reduces overfitting by limiting the model's capacity. The L2 penalty is smooth and differentiable, making it easy to optimize with gradient descent.
Real-World Applications: Preventing overfitting (the most common regularization technique in deep learning, known as weight decay), ridge regression (used in linear models when features are correlated), and image classification (added to the loss function of CNNs to improve generalization).
5.3 Elastic Net Regularization
Elastic Net combines L1 and L2 regularization:
Elastic Net balances the sparsity of L1 with the smoothness of L2. It is particularly useful when there are correlated features.
Mathematical properties: Elastic Net combines the advantages of L1 and L2 regularization. It encourages sparsity like L1 and stability like L2. The two parameters λ₁ and λ₂ control the balance between the L1 and L2 penalties.
Real-World Applications: High-dimensional regression (when there are many correlated features), feature selection (combines L1 sparsity with L2 stability), and genomic data analysis (selecting a subset of genes while accounting for correlations).
5.4 Dropout as a Regularization Loss
Dropout is a regularization technique that randomly drops a fraction of the neurons during training. This prevents co-adaptation of neurons and encourages the network to learn redundant representations. Dropout can be interpreted as an ensemble of many sub-networks.
Mathematical properties: Dropout is a stochastic regularization technique that has become a standard component in deep learning architectures. During training, each neuron is dropped with probability p, and the remaining neurons are scaled to maintain the expected activation. During inference, all neurons are used. Dropout is effective at preventing overfitting and is used in many state-of-the-art models.
Real-World Applications: All modern deep learning architectures (ResNet, EfficientNet, Transformers) use dropout for regularization. Dropout is also used as a form of Bayesian approximation for uncertainty estimation.
Applications
- Feature selection (L1).
- Preventing overfitting (L2, Dropout).
- High-dimensional regression (Elastic Net).
Strengths and limitations
| Strengths | Limitations |
|---|---|
| L1 provides feature selection. | L1 is non-differentiable at zero. |
| L2 is smooth and stable. | L2 does not perform feature selection. |
| Elastic Net combines both. | Elastic Net has two hyperparameters. |
| Dropout is computationally efficient. | Dropout requires careful tuning of p. |
Table 7: Strengths and limitations of regularization losses. L1 provides feature selection by driving weights to zero, but it is non‑differentiable at zero. L2 is smooth and stable but does not perform feature selection. Elastic Net combines both but introduces two hyperparameters. Dropout is computationally efficient yet requires careful tuning of the drop probability p.