Cost Functions and Distances - 1. Regression Losses: Continuous Targets

1. Regression Losses: Continuous Targets

Regression losses are used when the target variable is continuous (e.g., house prices, temperature, stock returns). The goal is to minimize the difference between the predicted value and the true value. The most common regression losses are the Mean Squared Error (MSE), the Mean Absolute Error (MAE), and the Huber Loss, which combines the best of both. The choice among these depends on the distribution of the target variable and the sensitivity to outliers.

The diagram in Fig 3 compares the behavior of MSE, MAE, and Huber loss as a function of the prediction error. MSE grows quadratically, heavily penalizing large errors. MAE grows linearly, making it more robust to outliers. Huber loss is quadratic for small errors and linear for large errors, providing a smooth transition between the two regimes.

This figure illustrates the key differences between the three most common regression losses. The Mean Squared Error (MSE) curve (blue) is a parabola: as the prediction error grows, the loss grows quadratically. This means that a single large error (outlier) can dominate the total loss and pull the model significantly off course. The Mean Absolute Error (MAE) curve (orange) is V-shaped, growing linearly with the error. This makes MAE much more robust to outliers, as a large error contributes only linearly to the total loss. The Huber loss (green) combines both: it is quadratic for small errors (providing smooth gradients) and linear for large errors (providing robustness). The transition point is controlled by the hyperparameter δ, which is shown here as 1.0. The choice among these losses depends on the presence of outliers in the data and the desired sensitivity to large errors.

1.1 Mean Squared Error (MSE)

The Mean Squared Error (MSE) is defined as:

It is the most widely used regression loss. MSE is the average of the squared differences between the predicted and true values. It is differentiable, convex, and has a well-defined gradient.

Mathematical properties: MSE is the maximum likelihood estimator for the parameters of a linear model when the errors are assumed to be normally distributed (Gaussian). It is also the minimum-variance unbiased estimator for the mean of a Gaussian distribution. The quadratic penalty means that MSE is sensitive to outliers; a single point with a large error can dominate the loss. This can be beneficial when large errors are particularly undesirable (e.g., in financial forecasting, where a large prediction error can lead to significant losses), but it can also cause the model to overfit to outliers. MSE is also used in the training of autoencoders and many other deep learning models, often combined with regularization to prevent overfitting.

Real-World Applications: Linear regression, house price prediction, weather forecasting, and corrosion rate prediction. MSE is used as the loss function in models for predicting temperature, precipitation, and marine heatwaves. In corrosion engineering, MSE is used to train neural networks for predicting corrosion rates (Wang et al., 2020). In wireless communications, the Normalized Mean Squared Error (NMSE) is used as a loss function for training neural networks in signal processing applications.

1.2 Mean Absolute Error (MAE)

The Mean Absolute Error (MAE) is defined as:

MAE is the average of the absolute differences between predictions and targets. It is less sensitive to outliers than MSE because the gradient is constant. This constant gradient means that MAE is more robust to outliers but has a discontinuity at zero, which can cause instability in gradient-based optimization.

Mathematical properties: MAE is the maximum likelihood estimator for the parameters of a linear model when the errors are assumed to follow a Laplace distribution. The median is the optimal predictor under MAE, making it a natural choice when the data contains many outliers. However, the non-differentiability at zero can be problematic for optimization; sub-gradients are often used instead. In practice, MAE is preferred when the data contains significant outliers and the goal is to produce a robust model that is not overly influenced by extreme values.

Real-World Applications: Radiation therapy dose prediction, financial forecasting, and robust regression modeling. DeepDoseNet, a deep learning model for 3D dose prediction in radiation therapy, uses MAE as its loss function to achieve accurate and robust predictions (Fan et al., 2019). MAE is also used in TensorFlow and Keras models for evaluating performance on real-world regression data with outliers.

1.3 Huber Loss

The Huber Loss (also called the Smooth L1 Loss) combines the best of MSE and MAE. It is quadratic for small errors and linear for large errors:

The hyperparameter δ controls the transition point between the quadratic and linear regimes. Huber loss is differentiable everywhere, making it suitable for gradient-based optimization. It is less sensitive to outliers than MSE but provides a smoother gradient than MAE.

Mathematical properties: The Huber loss is a robust estimator that combines the advantages of MSE and MAE. For errors smaller than δ, it behaves like MSE, providing a smooth gradient and quadratic convergence. For errors larger than δ, it behaves like MAE, providing robustness to outliers. The transition is smooth because the derivative is continuous at δ. The choice of δ is critical: a small δ makes the loss behave more like MAE, while a large δ makes it behave more like MSE. Huber loss is often used in regression tasks where outliers are expected but the model should still be sensitive to small errors.

Real-World Applications: Parameter estimation in nonlinear systems, robust regression, and Boston housing price prediction. Huber loss-guided neural networks are used for parameter estimation, effectively uncovering complex relationships in nonlinear systems (Chen et al., 2019). In time series trend filtering, Huber loss is adopted to suppress outliers while capturing both slow and abrupt trend changes. It is also used with Keras to create regression models robust to outliers on the Boston Housing dataset.

1.4 Log-Cosh Loss

The Log-Cosh Loss is another smooth approximation to MAE that is less sensitive to outliers than MSE:

For large errors, log-cosh behaves like MAE. For small errors, it behaves like MSE. It is twice differentiable and provides a smooth, robust alternative to Huber loss.

Mathematical properties: The log-cosh function is the logarithm of the hyperbolic cosine of the error. It is a smooth, convex function that is approximately quadratic near zero and approximately linear for large errors. Unlike Huber loss, which has a piecewise definition, log-cosh is a single, smooth function that is twice differentiable everywhere. This makes it easier to use in automatic differentiation frameworks and can lead to more stable optimization. The log-cosh loss is often used in regression tasks where robustness to outliers is desired but a smoother gradient than Huber is preferred.

Real-World Applications: Regression tasks with outliers, deep learning applications where a twice-differentiable robust loss is needed, and time-series forecasting with heavy-tailed error distributions.

1.5 Quantile Loss

The Quantile Loss (also called the Pinball Loss) is used for quantile regression, where the goal is to predict a specific quantile of the conditional distribution:

where $q\in [0,1]$ is the desired quantile. For $q=0.5$, it reduces to MAE. For $q=0.9$, over-predictions are penalized more heavily, making the model conservative.

Mathematical properties: Quantile loss is a linear loss function that is asymmetric when q ≠ 0.5. It is used to estimate the conditional quantiles of the target distribution, which provides a more complete picture of the uncertainty than a point estimate. For example, in financial risk management, predicting the 95th percentile of a loss distribution is crucial for Value-at-Risk (VaR) calculations. Quantile loss is also used in ensemble methods and in the construction of prediction intervals.

Real-World Applications: Financial risk management (Value-at-Risk), uncertainty quantification, and ensemble methods. Quantile loss is used in quantile regression forests and gradient boosting to construct prediction intervals and estimate the uncertainty of predictions in domains like energy forecasting and climate modeling.

Applications

• House price prediction (MSE).
• Robust regression with outliers (MAE, Huber).
• Quantile regression for risk management (Quantile Loss).
• Time-series forecasting (Log-Cosh).
• Radiation therapy dose prediction (MAE).

Strengths and limitations

Table 3: Strengths and limitations of regression losses. MSE is smooth and convex but highly sensitive to outliers. MAE is robust to outliers but non‑differentiable at zero. Huber combines robustness with smoothness but requires tuning the δ hyperparameter. Quantile loss provides distributional information but is asymmetric for q ≠ 0.5. The choice depends on the presence of outliers and the need for smooth gradients.