12. Contrastive and Self-Supervised Losses
Contrastive and self-supervised losses are used in unsupervised and self-supervised learning, where the goal is to learn representations from unlabeled data. These losses encourage the model to learn representations that are invariant to augmentations and discriminative between different instances.
12.1 InfoNCE Loss (SimCLR)
The InfoNCE Loss (used in SimCLR) is a contrastive loss that encourages the model to learn representations that are similar for positive pairs and dissimilar for negative pairs.
Real-World Applications: SimCLR self-supervised learning (pretraining encoders on unlabeled image datasets), contrastive learning (used in MoCo, CLIP, and other contrastive learning methods), and wearable time series (applied to high-dimensional health signals for downstream clinical classification) (Chen et al., 2020).
12.2 BYOL (Bootstrap Your Own Latent)
BYOL (Bootstrap Your Own Latent) is a self-supervised learning method that does not require negative samples.
Real-World Applications: Self-supervised learning (achieving state-of-the-art performance without requiring negative samples), image classification (pretraining on ImageNet), and object detection (pretrained BYOL encoders are used as backbones for detection models) (Grill et al., 2020).
12.3 MoCo (Momentum Contrast)
MoCo (Momentum Contrast) is a contrastive learning method that uses a momentum encoder and a queue of negative samples.
Real-World Applications: Contrastive learning (MoCo uses a momentum encoder and a queue of negative samples for efficient contrastive learning), image classification (pretraining on large unlabeled datasets), and transfer learning (learned representations transfer well to downstream tasks) (He et al., 2020).
Applications
- Self-supervised learning (SimCLR, BYOL, MoCo).
- Image representation learning (InfoNCE).
- Cross-modal retrieval (CLIP).
Strengths and limitations
| Strengths | Limitations |
|---|---|
| Learn representations without labels. | Require large batch sizes or queues. |
| BYOL does not require negative samples. | BYOL can be sensitive to architecture. |
| Scalable to large datasets. | Requires careful tuning. |
Table 14: Strengths and limitations of contrastive losses. They learn powerful representations without labels but often require large batch sizes or memory queues. BYOL avoids negative samples entirely, yet can be sensitive to architectural choices (e.g., predictor network, momentum coefficient). All methods scale to large datasets but demand extensive tuning of augmentations and hyperparameters.