Neural Networks Taxonomy - 9. Siamese and Metric Learning Networks

9. Siamese & Metric Learning Networks

A Siamese network is less a new layer than a new training setup: two (or three) identical towers with shared weights encode their inputs into a common space, and the network is trained so that similar inputs land close together and dissimilar ones far apart, as in Fig 11. The towers themselves are usually convolutional or Transformer encoders; what makes the family distinct is the metric-learning objective.

Origins and rise

The design dates to Bromley and colleagues' signature-verification network in 1993,^[118] and reached maturity with FaceNet, whose triplet loss learned face embeddings good enough for verification at scale.^[119] The same contrastive idea now powers self-supervised pre-training and cross-modal models such as the vision-language systems later in this tree.

Contrastive losses: pairs, triplets, and structure

What sets the family apart is the loss, and its history is a march toward using more of each batch. The original contrastive loss operated on pairs, pulling matching examples together and pushing non-matching ones apart beyond a fixed margin.^[120] The triplet loss popularized by FaceNet compares an anchor against one positive and one negative at once, sidestepping the need to calibrate an absolute distance.^[119] Because most triplets become uninformative once training gets going, structured losses such as lifted structured embedding use every positive and negative pair within a mini-batch, folding hard-negative mining into the objective rather than a preprocessing step.^[121]

Metric-based few-shot learning

A natural use of a learned metric is recognizing new classes from a handful of examples. Matching Networks cast one-shot classification as a differentiable nearest-neighbour lookup over a small support set, trained episodically to mirror the test conditions.^[122] Prototypical Networks pared this down to a single prototype per class — the mean of its support embeddings — and classify by distance to those prototypes.^[123] Relation Networks go a step further and learn the comparison itself, replacing a fixed distance with a small network that scores how well a query matches each class.^[124] The optimization-based cousins of these methods sit with the meta-learning family later in this tree.

Self-supervised representation learning

The same twin-tower recipe, applied without labels, became an engine of modern representation learning. SimCLR showed that strong augmentation, a projection head, and a large batch of in-batch negatives suffice to learn features that rival supervised pre-training.^[125] MoCo removed the need for huge batches with a momentum-updated encoder and a queue of negatives.^[126] Then came a surprise: BYOL and SimSiam dropped negatives entirely, avoiding collapse with a predictor and a stop-gradient instead of contrastive pushing.^[127][128] SwAV swapped explicit pairs for online cluster-assignment matching,^[129] while Barlow Twins and VICReg prevent collapse by regularizing the statistics of the embeddings — decorrelating feature dimensions, or controlling their variance and covariance.^[130][131] On vision Transformers the same instinct surfaces as the self-distillation of DINO.^[64]

Deep metric learning for retrieval

For search and face recognition the goal is an embedding whose distances rank correctly, and two strands dominate. Margin-softmax losses reshape ordinary classification into an angular metric: ArcFace, alongside its sibling CosFace, adds a fixed margin on the hypersphere so that same-identity embeddings cluster tightly, setting the standard for face recognition.^[132] The other strand refines pairwise training: proxy methods such as Proxy Anchor learn a few representative points to compare against, cutting the cost of mining real pairs,^[133] while Multi-Similarity and Circle Loss weight each pair by how informative it is, unifying many earlier losses under a single view of pair-similarity optimization.^[134][135]

Applications

• Face and signature verification, and biometric matching.
• One-shot and few-shot recognition.
• Image retrieval and self-supervised representation learning.

Strengths and limitations