Cost Functions and Distances - 7. Imbalanced Classification Losses

7. Imbalanced Classification Losses

Imbalanced classification losses are designed to address the problem of class imbalance, where one class (the minority class) has significantly fewer samples than the other classes. Standard losses like cross-entropy tend to ignore the minority class, leading to poor performance. The most common imbalanced losses are Weighted Cross-Entropy, Focal Loss, and Class-Balanced Loss.

7.1 Weighted Cross-Entropy

Weighted Cross-Entropy assigns a higher weight to the minority class during training. The loss is:

where $w_{y_i}$ is the weight assigned to class $y_i$. The weights are typically set inversely proportional to the class frequencies.

Mathematical properties: Weighted cross-entropy is a simple and effective way to handle class imbalance. By assigning higher weights to the minority class, the loss encourages the model to pay more attention to these samples.

Real-World Applications: Medical diagnosis (assigning higher weights to rare diseases), fraud detection (weighting fraudulent transactions more heavily), and anomaly detection (used in any task with imbalanced classes).

7.2 Focal Loss

Focal Loss was introduced in Section 2.6. It is designed to address class imbalance by down-weighting easy examples and focusing on hard examples.

Real-World Applications: RetinaNet object detection (focal loss was specifically designed for RetinaNet), imbalanced classification (medical diagnosis, fraud detection), and white blood cell detection (γ = 2.0 delivers the highest precision). RetinaNet with focal loss matches the speed of one-stage detectors while surpassing the accuracy of two-stage detectors (Lin et al., 2017).

7.3 Class-Balanced Loss

The Class-Balanced Loss is a general framework for re-weighting the loss based on the effective number of samples per class. The effective number is defined as:

where n is the number of samples in the class, and β is a hyperparameter that controls the rate at which the effective number saturates. The loss is then weighted by the inverse of the effective number.

Mathematical properties: The class-balanced loss is a more principled approach to handling class imbalance than simple weighting. It accounts for the diminishing returns of adding more samples to a class.

Real-World Applications: Imbalanced datasets (a principled approach to re-weighting), object detection (used with focal loss to further improve performance), and medical imaging (combined with other losses for rare disease detection).

7.4 Lovász Hinge Loss

The Lovász Hinge Loss is a loss function designed for semantic segmentation that directly optimizes the Intersection over Union (IoU) metric. It is defined using the Lovász extension of the hinge loss, which provides a smooth approximation of the IoU.

Mathematical properties: The Lovász hinge loss is a powerful loss for segmentation tasks where the goal is to maximize the IoU. Unlike cross-entropy, which treats each pixel independently, the Lovász hinge loss considers the entire segmentation mask and optimizes the IoU directly.

Real-World Applications: Semantic segmentation, medical image segmentation (used in state-of-the-art segmentation models), and autonomous driving (improving segmentation of road scenes).

Applications

• Medical diagnosis (Weighted Cross-Entropy).
• Object detection (Focal Loss).
• Semantic segmentation (Lovász Hinge Loss).
• Fraud detection (Class-Balanced Loss).

Strengths and limitations

Table 9: Strengths and limitations of imbalanced losses. Weighted cross‑entropy is simple but requires careful weight assignment. Focal loss is robust and effective yet demands tuning of γ. Class‑balanced loss is principled via effective numbers but introduces the β hyperparameter. Lovász hinge directly optimises IoU for segmentation, though it is computationally expensive.