Cost Functions and Distances - 3. Probabilistic Losses

3. Probabilistic Losses: Distributional Modeling

Probabilistic losses are used when the model outputs a probability distribution over the target variable, rather than a point estimate. These losses are derived from information theory and measure the divergence between the predicted distribution and the true distribution. The most common probabilistic loss is the Kullback-Leibler Divergence (KL Divergence), which measures the information loss when using the predicted distribution to approximate the true distribution.

3.1 Kullback-Leibler Divergence

The Kullback-Leibler Divergence (KL Divergence) measures the dissimilarity between two probability distributions P and Q:

KL Divergence is asymmetric and is non-negative, with equality only when P = Q.

Mathematical properties: KL Divergence is a fundamental concept in information theory and is the basis for many probabilistic loss functions. It measures the amount of information lost when Q is used to approximate P. In machine learning, KL Divergence is often used as a regularization term in variational autoencoders (VAEs) to encourage the latent distribution to be close to a prior distribution. It is also used in knowledge distillation.

Real-World Applications: Variational autoencoders (VAEs), knowledge distillation, transfer learning (KL divergence is used as a regularization loss to guide model fine-tuning), and drift detection (autoencoder-based models incorporate KL divergence to detect changes in data distribution). In knowledge distillation, the student model is trained to minimize the KL divergence between its output distribution and that of the teacher model (Hinton et al., 2015).

3.2 Jensen-Shannon Divergence

The Jensen-Shannon Divergence (JSD) is a symmetric version of KL Divergence:

where $M=(P+Q)/2$. JSD is symmetric, non-negative, and bounded between 0 and log(2).

Mathematical properties: Jensen-Shannon Divergence is a symmetric and smoothed version of KL Divergence. It is bounded between 0 and log(2), which makes it more stable and interpretable than KL Divergence. JSD is used in a variety of applications, including generative adversarial networks (GANs).

Real-World Applications: Generative adversarial networks (GANs), clustering, and model comparison. In GANs, JSD is used to measure the similarity between the generated distribution and the real data distribution (Goodfellow et al., 2014). It is also used in clustering to measure the distance between clusters.

3.3 Wasserstein Loss (Earth Mover's Distance)

The Wasserstein Loss (also called Earth Mover's Distance or Wasserstein Distance) measures the minimum cost of transforming one probability distribution into another. For two distributions P and Q, the 1-Wasserstein distance is:

Wasserstein loss is smooth and provides meaningful gradients even when the distributions have disjoint support.

Mathematical properties: Wasserstein loss is based on optimal transport theory and provides a geometrically meaningful distance between probability distributions. Unlike KL Divergence, which is undefined when the distributions have disjoint support, Wasserstein loss is well-defined for any pair of distributions and is smooth, making it ideal for training generative models.

Real-World Applications: Wasserstein GAN (WGAN) uses the 1-Wasserstein distance for more stable training (Arjovsky et al., 2017), IIoT anomaly detection (WGAN is used to generate high-fidelity minority samples), domain adaptation, and image retrieval. In the Industrial Internet of Things (IIoT), WGAN with Wasserstein distance is used for anomaly detection (Li et al., 2021).

3.4 Negative Log-Likelihood (NLL)

The Negative Log-Likelihood (NLL) is a general-purpose loss for probabilistic models. For a model that outputs a probability distribution over the target variable, the NLL is:

NLL is the negative logarithm of the likelihood of the data given the model parameters. Minimizing NLL is equivalent to maximizing the likelihood.

Mathematical properties: NLL is the foundational loss for all maximum likelihood estimation (MLE) methods. It is a general framework that encompasses many other loss functions as special cases. For example, when the model assumes a Gaussian distribution for the target, NLL reduces to MSE. When the model assumes a Bernoulli distribution, NLL reduces to binary cross-entropy. NLL is the default training objective for many deep learning models, including language models.

Real-World Applications: Language modeling (GPT, LLaMA), maximum likelihood estimation, and speech recognition (acoustic models with CTC loss). NLL is the standard loss for training autoregressive language models.

Applications

• Variational autoencoders (KL Divergence).
• Generative adversarial networks (Wasserstein Loss).
• Language modeling (Negative Log-Likelihood).
• Knowledge distillation (KL Divergence).

Strengths and limitations

Table 5: Strengths and limitations of probabilistic losses. KL divergence has a strong information‑theoretic foundation but is asymmetric. JSD is symmetric and bounded, yet computationally more expensive. Wasserstein loss provides meaningful gradients even for disjoint supports, but exact computation is costly. NLL is a general‑purpose framework but can be sensitive to model misspecification.