Neural Networks Taxonomy - 23. HyperNetworks and Parameter-Efficient Tuning

23. HyperNetworks & Parameter-Efficient Tuning

Two related ideas treat a network's weights as something to generate or nudge cheaply. A hypernetwork is one network that outputs the weights of another. Parameter-efficient fine-tuning freezes a large pretrained model and trains a tiny add-on; LoRA, in Fig 25, adds a low-rank update B·A beside each frozen weight, so only a few million parameters are learned.

Origins and rise

HyperNetworks introduced the idea of one network generating another's weights.^[315] LoRA made low-rank adaptation the default way to fine-tune large language models,^[316] and QLoRA combined it with 4-bit quantisation to fine-tune very large models on a single GPU.^[317]

Weight-generating hypernetworks

The original HyperNetworks idea let an embedding condition the weights of a recurrent network, compressing many parameter sets into one generator.^[315] A related trick, dynamic convolution, generates several convolution kernels and mixes them by input-dependent attention, keeping a model compact yet expressive at runtime.^[318] The same generator-as-controller pattern underpins the model-based meta-learning networks that condition a learner on task context.

Adapter modules and bottleneck insertion

Adapters add small trainable modules between the layers of a frozen Transformer, the family sketched in Fig 25. The Houlsby adapter inserted a bottleneck module after every sub-layer and matched full fine-tuning with only a few percent of the parameters,^[319] while the Pfeiffer adapter showed that placing adapters only after the feed-forward block is almost as effective and more efficient. This spawned a long line of cross-task and cross-lingual adapter stacks that share one backbone.

Low-rank reparameterisation

LoRA reframed adaptation as a low-rank update B·A added to each frozen weight, trainable yet mergeable back into the base,^[316] and QLoRA stacked 4-bit quantisation underneath to fine-tune 65-billion-parameter models on a single GPU.^[317] AdaLoRA made the rank itself adaptive, allocating parameter budget to the layers that need it most,^[320] and DyLoRA trained a range of ranks at once so the right one can be picked at inference.

Prompt-style and lightweight tuning

A second branch leaves the weights untouched and tunes only continuous inputs. Prefix-Tuning learned key-value vectors prepended to each attention layer,^[321] Prompt Tuning showed that tuning just the input embedding is enough at very large scale,^[322] and P-Tuning v2 extended prefix-style prompts to smaller models and harder tasks, closing the gap to fine-tuning.^[323] IA³ is the extreme of the spectrum — scaling each key, value, and feed-forward vector by a single learned number per layer,^[324] showing that a few thousand parameters can already steer a large model.

Applications

• Cheap fine-tuning of foundation models.
• Per-task or per-user adapters over one shared backbone.
• Conditional weight generation.

Strengths and limitations