Neural Networks Taxonomy - 4. Transformer Networks

4. Transformer Networks

The Transformer replaces recurrence with self-attention: every element of a sequence looks directly at every other element and decides how much each one matters. Because those comparisons happen all at once rather than step by step, the whole sequence is processed in parallel — the exact bottleneck that slowed recurrent networks down. The architecture stacks multi-head attention and feedforward blocks into an encoder and a decoder, wired together as in Fig 6. At its core, the attention mechanism computes a weighted sum of values based on the similarity between queries and keys: Attention(Q,K,V) = softmax(QK^T / sqrt(d_k)) V. The scaling factor sqrt(d_k) prevents dot-products from growing too large and pushing the softmax into saturation, which stabilises gradients. The multi-head formulation runs this computation several times in parallel, with different learned projections, allowing the model to attend to different aspects of the input simultaneously.

Origins and rise

The design was introduced in 2017 in the paper “Attention Is All You Need,” which showed that attention alone, with no recurrence or convolution, set a new state of the art in machine translation while training far faster.^[47] Two directions followed almost immediately. BERT used only the encoder, pre-trained to fill in masked words, and became the backbone of search and classification.^[48] The GPT line used only the decoder to predict the next token, and scaling it to hundreds of billions of parameters produced the general-purpose language models behind today's assistants.^[49] Attention then crossed into vision: the Vision Transformer (ViT) cut an image into patches and treated them as a sequence, matching or beating convolutional networks given enough data.^[50]

4.1 The base architecture

The original 2017 design is an encoder–decoder built from two repeated pieces: multi-head self-attention and a position-wise feed-forward network, each wrapped in a residual connection and layer normalisation.^[47] Attention itself is a scaled dot-product: every token emits a query, a key, and a value, and each output is a weighted average of values whose weights come from query–key similarity. Running several heads in parallel lets one layer attend to different relationships at once. Because attention is order-blind, the input is tagged with positional encodings so the network can still tell first token from last. Almost every model below keeps this skeleton and changes only which half it uses, how attention is computed, or what the network is trained on.

4.2 Natural-language models

Keeping only the encoder gives a bidirectional model that reads a whole passage at once, suited to understanding rather than generation. BERT set the template, pre-training on masked-word prediction before fine-tuning for search, classification, and question answering.^[48] RoBERTa showed the recipe was under-trained and won large gains from more data and longer schedules alone;^[51] ELECTRA swapped masked-word filling for a more sample-efficient “spot the replaced token” objective;^[52] and DeBERTa's disentangled attention, which encodes content and position separately, pushed the family past human baselines on several benchmarks.^[53] ALBERT, DistilBERT, and Megatron-BERT trade accuracy against size and speed.

Keeping only the decoder gives an autoregressive model that predicts the next token, and this is the branch that became today's large language models. Scaling GPT to hundreds of billions of parameters revealed that a single next-token objective, given enough data, yields few-shot learning on tasks it was never explicitly trained on.^[49] The Chinchilla study then reset the field's scaling laws, showing that most large models were badly under-trained and that data and parameters should grow together.^[54] Google's PaLM scaled the idea to 540 billion parameters,^[55] while Meta's LLaMA proved that far smaller, carefully trained open models could rival them,^[56] seeding an open ecosystem of Mistral, Falcon, and Gemma — and the mixture-of-experts Mixtral, which sparsely routes each token to a few expert sub-networks to grow capacity without a matching rise in compute.^[57]

Keeping both halves suits sequence-to-sequence tasks such as translation and summarisation. T5 cast every language problem as text-to-text, so one model and one objective could translate, summarise, and answer questions.^[58] BART pre-trained by corrupting text and learning to reconstruct it, excelling at generation and summarisation;^[59] multilingual and instruction-tuned variants — mT5, Flan-T5, mBART, Pegasus, and ProphetNet — extend the same encoder–decoder core.

4.3 Vision transformers

Attention reached images by slicing a picture into fixed-size patches and feeding them as a sequence — the Vision Transformer, which matched convolutional networks once given enough data.^[50] DeiT removed the giant-dataset requirement with distillation and stronger augmentation, training a competitive ViT on ImageNet alone.^[60] Swin Transformer reintroduced a convolution-like hierarchy, computing attention within shifted local windows so cost grows linearly with image size, which made transformers practical backbones for detection and segmentation.^[61] A parallel thread pursued self-supervised pre-training: BEiT predicted masked visual tokens in the style of BERT,^[62] the masked autoencoder (MAE) reconstructed heavily masked pixels and made large-scale pre-training cheap,^[63] and DINO showed that self-distillation makes ViT features segment objects with no labels at all, later scaled up as DINOv2.^[64] PVT, CvT, Twins, PoolFormer, CaiT, CrossViT, and iBOT fill in the design space between these poles.

4.4 Efficient and long-context transformers

Self-attention's cost grows with the square of sequence length, so a whole subfamily attacks that quadratic wall. Sparse-attention models let each token see only a subset of the others: Longformer combines a sliding window with a few global tokens,^[65] and BigBird adds random links while proving it retains the full model's expressive power.^[66] Reformer groups similar tokens with locality-sensitive hashing,^[67] while Linformer and Performer approximate attention with low-rank or kernel-based maps that make it linear.^[68] A different tack keeps attention exact but rewrites its implementation: FlashAttention reorders the computation to minimise reads and writes to GPU memory, giving large speed-ups and far longer contexts with identical results.^[69]

4.6 Decision and generalist transformers

The same sequence model can drive action as well as language. The Decision Transformer reframes reinforcement learning as sequence modelling: conditioned on a desired return, it simply predicts the next action, sidestepping explicit value functions and bootstrapping.^[70] Gato pushed the generalist idea furthest, training one transformer to caption images, hold a conversation, stack blocks with a robot arm, and play Atari from a single set of weights.^[71] Both point toward the multimodal systems of family 15, where a shared transformer backbone fuses text, images, audio, and action.

Applications

• Large language models, translation, summarisation, and code generation.
• Vision, speech, and multimodal models that mix text with images or audio.
• Protein and molecule modelling, and increasingly reinforcement learning and robotics.

Strengths and limitations