Cost Functions and Distances - 2. Classification Losses: Discrete Targets

2. Classification Losses: Discrete Targets

Classification losses are used when the target variable is discrete (e.g., class labels). The goal is to minimize the discrepancy between the predicted class probabilities and the true class labels. The most common classification loss is Cross-Entropy Loss (also called Log Loss), which measures the dissimilarity between the predicted probability distribution and the true distribution. Other classification losses include Hinge Loss (used in Support Vector Machines) and Focal Loss (designed to address class imbalance).

The diagram in Fig 4 illustrates the behavior of cross-entropy loss as a function of the predicted probability for a binary classification problem. When the true label is 1, the loss approaches zero as the predicted probability approaches 1, and increases to infinity as the predicted probability approaches 0. This logarithmic penalty encourages the model to make confident predictions.

This figure demonstrates the behavior of cross-entropy loss for binary classification. When the true label is 1, the loss is -log(ŷ). As the predicted probability ŷ approaches 1 (the model is very confident and correct), the loss approaches 0. As ŷ approaches 0 (the model is very confident but wrong), the loss approaches infinity. This logarithmic penalty strongly discourages confident but incorrect predictions. The loss at ŷ = 0.5 (random guessing) is 0.69, and at ŷ = 0.1 it is 2.30. This asymmetric penalty is what makes cross-entropy the default choice for classification tasks, as it pushes the model to make calibrated probability estimates.

2.1 Binary Cross-Entropy Loss

The Binary Cross-Entropy Loss is used for binary classification (two classes). It is defined as:

where $y_i\in \{0,1\}$ and ${\hat{y}}_i\in [0,1]$ is the predicted probability of the positive class.

Mathematical properties: Binary cross-entropy is a convex function of the logits (pre-activation values) and is the standard loss for binary classification in neural networks. It is derived from the maximum likelihood principle: the model outputs a probability, and the loss measures how well this probability matches the observed binary label. The gradient with respect to the logits is simple and well-behaved, making it compatible with gradient-based optimization.

Real-World Applications: Logistic regression, spam detection, medical diagnosis (chest X-ray classification), and YOLO object detection (used for objectness loss and classification loss). Binary cross-entropy is the standard loss for training models to detect masses in chest X-rays and for classifying emails as spam or not spam.

2.2 Categorical Cross-Entropy Loss

The Categorical Cross-Entropy Loss generalizes binary cross-entropy to multi-class classification (more than two classes). It is defined as:

where $y_{ic}$ is 1 if the sample belongs to class c, and 0 otherwise, and ${\hat{y}}_{ic}$ is the predicted probability for class c.

Mathematical properties: Categorical cross-entropy is the negative log likelihood of the multinomial distribution. It is the standard loss for multi-class classification in deep learning. The softmax activation ensures that the outputs sum to 1 and are positive, forming a valid probability distribution. The gradient with respect to the logits is simple and well-behaved, making it compatible with backpropagation.

Real-World Applications: Image classification (ResNet, EfficientNet, Vision Transformers on ImageNet), multi-class classification (digit recognition, species classification), and remote sensing (parcel-level crop classification and risk prioritization). Categorical cross-entropy is the default loss for training state-of-the-art image classification models.

2.3 Sparse Cross-Entropy Loss

The Sparse Categorical Cross-Entropy Loss is a variant of categorical cross-entropy that uses integer class labels instead of one-hot encoded vectors. The loss is the same as categorical cross-entropy, but the labels are provided as integers (0, 1, 2, ..., C-1) instead of one-hot vectors.

Mathematical properties: Sparse cross-entropy is equivalent to categorical cross-entropy but is more memory-efficient because it avoids storing the one-hot encoded matrix. This is particularly useful for large datasets with many classes. The gradient is the same as for categorical cross-entropy, and the loss function is convex with respect to the logits.

Real-World Applications: Large-scale classification (language modeling with large vocabularies), deep learning frameworks (TensorFlow and PyTorch default for multi-class classification with integer labels). Sparse cross-entropy is used in NLP tasks like language modeling where the vocabulary size can be hundreds of thousands of tokens.

2.4 Hinge Loss (SVM Loss)

The Hinge Loss is used in Support Vector Machines (SVMs) and is defined as:

where $y\in \{-1,1\}$ is the true label and $\hat{y}$ is the raw output (logit) of the model.

Mathematical properties: Hinge loss is convex but not differentiable at the point where $y\hat{y}=1$. Sub-gradients are used for optimization. The loss is designed to maximize the margin between classes, which leads to the well-known SVM formulation.

Real-World Applications: Support Vector Machines (SVMs), text classification (document categorization), and image classification (SVMs were popular before deep learning). Hinge loss with SVMs is used in linear classifiers for text categorization and in kernel-based methods for non-linear classification.

2.5 Squared Hinge Loss

The Squared Hinge Loss is a variant of hinge loss that squares the error:

Squared hinge loss is differentiable everywhere and is smoother than hinge loss. It penalizes margin violations quadratically.

Mathematical properties: Squared hinge loss is convex and smooth, which makes it easier to optimize than regular hinge loss. It is often used in Support Vector Machines with differentiable optimization algorithms. The quadratic penalty for large margin violations makes it more sensitive to outliers than the standard hinge loss.

Real-World Applications: Differentiable SVMs, kernel methods with smooth optimization, and deep learning applications where smooth gradients are desirable.

2.6 Focal Loss

The Focal Loss was introduced to address the class imbalance problem in object detection. It is defined as:

where $p_t$ is the predicted probability for the true class, and $\gamma \ge 0$ is a focusing parameter.

Mathematical properties: Focal loss is a dynamically scaled cross-entropy loss that down-weights the contribution of easy examples and focuses on hard examples. This makes it particularly effective for datasets with severe class imbalance. The focusing parameter γ controls the rate at which easy examples are down-weighted. A typical value for γ is 2.

Real-World Applications: RetinaNet object detection (focal loss was specifically designed for RetinaNet), imbalanced classification (medical diagnosis, fraud detection), and white blood cell detection (γ = 2.0 delivers the highest precision). RetinaNet with focal loss matches the speed of one-stage detectors while surpassing the accuracy of two-stage detectors (Lin et al., 2017).

Applications

• Image classification (Cross-Entropy).
• Object detection (Focal Loss).
• Support Vector Machines (Hinge Loss).
• Spam detection (Binary Cross-Entropy).

Strengths and limitations

Table 4: Strengths and limitations of classification losses. Cross‑entropy provides calibrated probabilities but can be over‑confident. Hinge loss maximizes the margin but does not output probabilities. Focal loss handles class imbalance effectively but requires tuning the focusing parameter γ. Squared hinge is smooth and differentiable yet remains sensitive to outliers.