Neural Networks Taxonomy - 12. Attention Plug-in Modules

12. Attention Plug-in Modules

Not every use of attention is a full Transformer. A family of lightweight attention modules exists to be bolted on to an existing convolutional network, letting it reweight its own features by importance. As Fig 14 shows, the module looks at a feature map, computes a set of weights, and multiplies them back in to emphasise the informative channels or locations.

Origins and rise

Squeeze-and-Excitation networks won the 2017 ImageNet challenge by recalibrating channels with a tiny attention branch,^[156] and the convolutional block attention module (CBAM) added spatial attention on top the following year.^[157] They are conceptual cousins of the Transformer's self-attention and of the neighbour weighting in graph attention.

Channel attention

The simplest question an attention module can ask is which channels matter. Squeeze-and-Excitation, introduced above, answers it by pooling each channel to a single number, learning a per-channel gate from that summary, and rescaling the feature map.^[156] ECA-Net shows the small fully-connected bottleneck SE uses to mix channels is unnecessary: a cheap one-dimensional convolution over neighbouring channels captures the same cross-channel interaction with a handful of extra parameters.^[158] Selective Kernel networks add a second axis of choice, attending over branches with different receptive-field sizes so the network adapts its effective kernel to the scale of the object in view.^[159]

Adding spatial attention

Channels say what; pixels say where. The convolutional block attention module (CBAM), noted above, chains a channel gate with a spatial one so the network learns both which features and which locations to emphasise.^[157] BAM arranges the same two attentions in parallel rather than in sequence and places the combined module at the network's bottlenecks, where downsampling makes feature selection most valuable.^[160]

Capturing long-range context

Convolution is local by construction, so a distant but relevant pixel influences another only after many layers. The non-local block borrows the Transformer's idea directly, computing the response at each position as a weighted sum over all positions and capturing long-range dependencies in a single step.^[161] Its cost is quadratic in the number of pixels, which spurred cheaper variants: GCNet observed that the non-local attention map is nearly identical for every query and collapsed it into a single shared context, fusing the thrift of Squeeze-and-Excitation with the reach of the non-local block.^[162] CCNet instead gathers context along criss-cross paths — each pixel attends only to its own row and column — and recovers full-image coverage by stacking two such passes, cutting the cost sharply for dense prediction.^[163]

Self-attention as a convolution replacement

A more radical step replaces convolution outright. Stand-alone self-attention swaps every spatial convolution in a residual network for a local self-attention layer, showing a vision backbone can be built from attention alone.^[164] Because full two-dimensional attention is expensive, axial attention factorises it into attention along rows followed by attention along columns; Axial-DeepLab builds a panoptic-segmentation backbone on this decomposition, reaching a large receptive field at manageable cost.^[165] These modules foreshadow the vision Transformer, which abandons the convolutional backbone entirely.

Applications

• Drop-in accuracy boosts for classification and detection backbones.
• Efficient mobile and embedded vision models.
• Any convolutional pipeline that benefits from sharper feature selection.

Strengths and limitations