Neural Networks Taxonomy - 8. Capsule Networks

8. Capsule Networks

A capsule network was proposed to fix a specific weakness of convolutional networks: pooling throws away where a feature is and how it is oriented. A capsule outputs a small vector instead of a single number, encoding a feature's pose (position, orientation, scale) alongside its presence. Lower capsules then send their output to whichever higher capsule they agree with, a process called routing by agreement, sketched in Fig 10.

Origins and rise

Sabour, Frosst, and Hinton introduced dynamic routing between capsules in 2017, with a matrix-capsule variant following soon after.^[113] The idea is elegant and viewpoint-robust on small datasets, but the routing procedure is slow and has not yet scaled to the large benchmarks where convolutional and Transformer models dominate, so it remains largely a research direction.

Routing by agreement

The defining question for the family is how a lower capsule decides which higher capsule should receive its output. The original CapsNet answered with dynamic routing: it measures the agreement between a capsule's prediction and each candidate parent and iteratively strengthens the couplings that agree, squashing each vector's length to encode how confident it is that the feature is present.^[113] Matrix capsules replaced this with a more principled scheme: each capsule carries a 4×4 pose matrix and a separate activation, and routing is cast as expectation-maximization clustering, with lower capsules acting as data points assigned to the higher capsules that behave like Gaussian clusters.^[114] EM routing made the geometry of pose explicit and improved robustness to viewpoint and adversarial changes, at the cost of a delicate procedure that is slow to train.

Deeper and more efficient capsules

Both routing schemes were first demonstrated on shallow networks over small images, and stacking many capsule layers proved hard because iterative routing is expensive and its gradients are noisy. DeepCaps tackled this by building a deeper capsule network around 3D convolutional routing, trimming the parameter count while pushing capsule accuracy past the original on CIFAR-scale benchmarks.^[115] It stands for a broader effort to keep the pose-aware inductive bias of capsules while borrowing the depth and efficiency that let convolutional and Transformer backbones scale.

Unsupervised and 3D capsules

Later work moved capsules beyond supervised image classification. Stacked capsule autoencoders turn the idea inside out: instead of routing by agreement they infer object capsules and their parts with no labels at all, reconstructing an image as a composition of part templates and reading out the discovered capsules for strong unsupervised classification.^[116] The pose machinery also transfers naturally to geometry: 3D point capsule networks apply capsules to unordered point clouds, learning latent capsules that capture the parts of a shape for reconstruction, interpolation, and part segmentation.^[117] Together these variants show the family is less a single architecture than a recurring bet that explicit part–whole structure is worth modelling.

Applications

• Viewpoint-robust recognition on small image datasets.
• Medical-imaging research where part-whole structure matters.
• A testbed for ideas about pose and equivariance.

Strengths and limitations