Neural Networks Taxonomy - 22. Meta-Learning and Few-Shot Architectures

22. Meta-Learning & Few-Shot Architectures

Meta-learning aims to “learn to learn”: produce a model that adapts to a brand-new task from only a handful of examples. Optimisation-based methods use the bi-level loop in Fig 24 — an inner loop adapts to each task, while an outer loop updates a shared initialisation so that adaptation is fast.

Origins and rise

MAML learned an initialisation from which a few gradient steps solve a new task,^[308] and Reptile reached similar results with a simpler first-order update.^[309] Metric-based methods such as Prototypical Networks instead classify by distance to per-class prototypes — the same metric-learning idea applied to few-shot problems.^[123]

Optimisation-based meta-learning

Optimisation-based methods, the family sketched in Fig 24, learn how to adapt rather than how to classify. MAML is the archetype: it learns an initialisation from which a few gradient steps on a new task reach a good solution,^[308] and Reptile reached the same end with a far simpler first-order update that avoids second-order derivatives.^[309] Meta-SGD goes further and meta-learns not just the initialisation but the learning rate and update direction for each parameter,^[310] while MetaOptNet replaces the inner gradient descent with a convex classifier such as a support-vector machine, casting few-shot learning as meta-learning a learner.^[311] When the model is too large to differentiate through directly, latent embedding optimisation (LEO) instead meta-learns in a low-dimensional latent space, recovering fast adaptation for big architectures.^[312]

Metric-based few-shot learning

Metric-based methods skip adaptation altogether and learn a comparison function, so a new class is recognised by its relation to a few labelled examples — an idea shared with the Siamese and metric-learning family. Matching Networks set the template by classifying a query with a soft nearest-neighbour vote over the support set,^[122] and Prototypical Networks simplified it to distance from a per-class prototype, the mean of its support embeddings.^[123] Relation Networks make that comparison itself learnable rather than fixing a Euclidean metric.^[124] A second thread leans on learned architectures rather than fixed metrics: the simple neural attentive meta-learner (SNAIL) embeds a temporal convolution and attention module that, given a short episode of labelled examples, learns to use them as context for the query.^[313]

Model-based meta-learning

Model-based methods encode the “how to learn” rule directly into the architecture — typically with external memory or weight generation. The memory-augmented neural network (MANN) stores labelled examples in an addressable external memory and reads from it at inference, so one-shot learning reduces to writing the new example and looking it up.^[314] More broadly, any HyperNetwork that generates a model's weights conditioned on the task can be read as model-based meta-learning — the idea formalised in the next family.^[315] These designs adapt in a single forward pass, which is faster than optimisation-based methods, but the learned strategy is less explicit and generalises less predictably to tasks unlike those seen in training.

Applications

• Few-shot image classification.
• Fast adaptation in robotics and personalisation.
• Learning initialisations and hyper-parameters.

Strengths and limitations