Neural Networks Taxonomy - 1. Feedforward Neural Networks

1. Feedforward Neural Networks (FNN)

The feedforward neural network is the family the introduction already described: units arranged in layers, information flowing strictly forward from input to output, with no loops. Its most common form is the multilayer perceptron (MLP), a stack of fully connected Dense layers each followed by a non-linearity, sketched in Fig 3. Everything else in this taxonomy can be read as an MLP with extra structure bolted on to suit a particular kind of data.

Origins and rise

The lineage starts with Rosenblatt's perceptron in 1958, a single trainable layer that could separate linearly separable classes.^[3] Its inability to solve even the XOR problem stalled the field until the 1986 popularisation of backpropagation by Rumelhart, Hinton, and Williams made it practical to train the hidden layers of a multilayer network.^[4] Soon after, the universal approximation theorem showed that a feedforward network with a single sufficiently wide hidden layer can approximate any continuous function to arbitrary accuracy^[5] — a reassuring result that says nothing about how to learn such a network efficiently, which is exactly what the specialised families that follow address.

1.1 Fully connected (dense)

Beyond the plain MLP, the fully connected branch holds several notable variants. The deep autoencoder stacks an encoder and a mirror-image decoder to squeeze data into a low-dimensional code and reconstruct it; Hinton and Salakhutdinov showed such a network could beat classical PCA at dimensionality reduction once its layers were pre-trained greedily.^[6] That same layer-by-layer pre-training defined the deep belief network (DBN), a stack of restricted Boltzmann machines that briefly made very deep networks trainable before ReLU activations and better initialisation made the ritual unnecessary.^[7] A different branch trades learning for speed: the extreme learning machine (ELM) fixes the hidden weights at random and solves only the output layer in closed form,^[8] an idea echoed by the random vector functional link (RVFL) network and the broad learning system (BLS), which widen the network rather than deepen it.

1.2 Partially and locally connected

Connecting every unit to every other is wasteful when the input has local structure. A locally connected layer keeps the spatial grid of an image but gives each position its own unshared weights — the halfway house between a dense layer and the weight-sharing convolution of the next family. The pioneering example was Fukushima's neocognitron (1980), a hierarchy of local feature detectors and pooling cells that fed directly into the design of convolutional networks.^[9]

1.3 Linear and shallow models

At the shallow end sit the single-layer models that started the field. Rosenblatt's perceptron and Widrow and Hoff's ADALINE (1960) are linear classifiers trained by simple weight updates; MADALINE wired several ADALINEs together into one of the first practical multi-layer trainers. Logistic regression is the same object seen from statistics — a one-layer network with a sigmoid output. All share the perceptron's ceiling: with no hidden layer they can only draw linear boundaries, the very wall that backpropagation was popularised to break.

Applications

• Tabular prediction: credit scoring, churn, and risk models over fixed-length feature vectors.
• The final classification or regression “head” on top of almost every other architecture in this article.
• Function approximation inside larger systems — value estimates in reinforcement learning, for example.

Strengths and limitations

That second limitation — blindness to structure — is precisely what the next family fixes. Feeding a million-pixel image into a dense layer needs an impossible number of weights; convolutional networks share weights across the image to make vision tractable.