Neural Networks Taxonomy - 2. Convolutional Neural Networks

2. Convolutional Neural Networks (CNN)

A convolutional neural network replaces full connectivity with a small learnable filter that slides across the image, reusing the same weights at every position. That single idea — weight sharing plus local receptive fields — slashes the parameter count and builds in translation invariance: a feature detector that finds an edge in one corner finds it everywhere. A typical CNN alternates conv layers with pool layers that shrink the spatial size, then finishes with a feedforward head, as in Fig 4. Mathematically, a 2D convolution computes a dot product between the filter and a local patch of the input at every location: (I * K)(i,j) = sum_m sum_n I(i+m, j+n) K(m,n). The translational equivariance of this operation is the reason a CNN can recognise a face regardless of where it appears in the frame.

Origins and rise

LeCun's LeNet-5 read handwritten digits for the postal service in the 1990s,^[10] but the family exploded in 2012 when AlexNet won the ILSVRC competition on the ImageNet dataset by a wide margin, using ReLU activations and two GPUs to train a then-enormous model.^[11][12] VGG then showed that stacking many small 3×3 filters worked better than a few large ones,^[13] but pushing depth further ran into the vanishing-gradient problem: error signals faded before they reached the early layers. ResNet solved it in 2015 with the residual connection — the dashed skip in Fig 4 — which adds a layer's input to its output so gradients have a short path back, making networks hundreds of layers deep trainable at last.^[14]

2.1 Image-classification backbones

The backbone zoo is the convolutional network's heartland. After AlexNet and VGG, GoogLeNet's Inception module ran filters of several sizes in parallel to widen a network cheaply,^[15] and ResNet's skip connection unlocked real depth. DenseNet took connectivity to its limit, feeding every layer's output to all later layers,^[16] while ResNeXt and Wide ResNet tuned the width-versus-depth trade-off. A parallel push shrank backbones to fit a phone: MobileNet swapped full convolutions for cheap depthwise-separable ones,^[17] and ShuffleNet, SqueezeNet, and GhostNet squeezed out more. EfficientNet balanced depth, width, and resolution with one compound scaling rule,^[18] a line that runs up through RegNet and RepVGG to ConvNeXt, which modernised a plain ResNet with Transformer-era training tricks to rival the vision transformers it was answering.^[19] The search for optimal backbones also spurred the Neural Architecture Search methods; EfficientNet's scaling rule, for instance, emerged from a systematic grid search over depth, width, and resolution, proving that balancing these three dimensions is far more effective than scaling any one in isolation.

2.2 Object detection

Detection adds where to what. The two-stage lineage — R-CNN, Fast R-CNN, and Faster R-CNN, which folded region proposals into the network itself^[20] — trades speed for accuracy, and Cascade and Sparse R-CNN refined it. One-stage detectors instead predict boxes directly for real-time speed: the YOLO family (v1 through v10, plus YOLOX and PP-YOLO),^[21] SSD, and RetinaNet, whose focal loss fixed the foreground-background imbalance that had capped one-stage accuracy.^[22] Anchor-free designs (FCOS, CenterNet, CornerNet) dropped the hand-tuned anchor boxes, and DETR recast detection as direct set prediction with a Transformer, removing non-maximum suppression from the pipeline^[23] — a bridge to the Transformer family.

2.3 Segmentation

Segmentation labels every pixel. The fully convolutional network (FCN) made this practical by turning the classification head into a dense one;^[24] U-Net's symmetric encoder-decoder with skip connections became the default for medical and scientific imaging,^[25] and the DeepLab series added atrous (dilated) convolutions and multi-scale pooling for scene parsing.^[26] Instance segmentation, which separates individual objects, is led by Mask R-CNN, which bolts a mask branch onto Faster R-CNN,^[27] alongside YOLACT and SOLO. Panoptic methods (Panoptic FPN, Mask2Former, OneFormer) unify the two, and the Segment Anything Model (SAM) turned segmentation into a promptable foundation task trained on a billion masks.^[28]

2.4 Video and 3D CNNs

Extending the convolution to time gives video models. C3D and I3D inflate 2D filters into 3D to learn motion, the latter bootstrapping from image-pretrained weights and the Kinetics dataset.^[29] R(2+1)D factorises space and time for efficiency, X3D scales a tiny network along several axes, and SlowFast runs two pathways — a slow one for appearance and a fast one for motion — that fuse for strong action recognition.^[30]

2.5 Restoration and super-resolution

The same machinery reconstructs images. SRCNN first showed a three-layer network could upscale images better than hand-crafted interpolation;^[31] EDSR and RCAN deepened it, and adversarial training brought photo-realistic texture — SRGAN and then ESRGAN, whose enhanced generator set the quality bar.^[32] Real-ESRGAN pushed it to real-world degradations, and Transformer-based restorers such as SwinIR and HAT now lead the field.^[33]

Applications

• Image classification, object detection, and semantic segmentation.
• Medical imaging, satellite and industrial inspection, and photo restoration.
• A feature extractor inside larger multimodal and generative systems.

Strengths and limitations

The limitation that a convolution only sees a local patch is one the Transformer later removed for vision, letting every location attend to every other. Lighter fixes also exist: attention modules reweight a convolutional network's own features, and architecture search can design the backbone automatically. For data that arrives as a sequence rather than a grid, though, the field first turned to a different design entirely.