Neural Networks Taxonomy - 14. Reinforcement Learning Architectures

14. Reinforcement Learning Network Architectures

Reinforcement learning (RL) is a training paradigm rather than a single architecture: an agent learns by trial and error to take actions that maximise long-term reward in an environment, the loop shown in Fig 16. The agent's policy and value functions are neural networks — typically feedforward or convolutional nets when the input is pixels.

Origins and rise

Deep RL arrived in 2015 when the deep Q-network (DQN) learned to play Atari games directly from pixels at human level.^[184] Policy-gradient methods such as PPO then made training more stable and became the workhorse for continuous control.^[185] The same machinery drives the reinforcement-learning-from-human-feedback step that aligns modern Transformer language models, and it powered the controllers behind early architecture search.

Value-based methods

The DQN template — a network that estimates the value of each action, trained on replayed experience^[184] — was sharpened by a rapid succession of fixes. Double DQN corrected the systematic over-estimation of action values,^[186] and the dueling architecture split the network into separate state-value and advantage streams.^[187] A parallel line replaced the single value estimate with a whole distribution over returns: C51 modelled that distribution on a fixed support,^[188] QR-DQN learned its quantiles directly,^[189] and IQN sampled them implicitly for a fuller picture of risk.^[190] Rainbow showed these gains are complementary, folding six of them into one agent,^[191] while Ape-X scaled learning across hundreds of actors feeding one prioritised replay buffer.^[192]

Policy-gradient and actor-critic methods

Where value-based agents suit discrete actions, policy-gradient methods optimise the policy directly and handle continuous control. A3C ran many actor-learners asynchronously to decorrelate their updates,^[193] and TRPO made large policy steps safe by constraining each update to a trust region^[194] — a constraint that PPO later approximated with a simple clipped objective.^[185] For continuous action spaces, DDPG carried the deterministic-policy idea into deep networks,^[195] TD3 curbed its value over-estimation with twin critics and delayed updates,^[196] and SAC added an entropy bonus that rewards exploration, becoming a robust default for robotics.^[197]

Model-based agents and world models

Model-free agents are notoriously sample-hungry, so a complementary strand learns a model of the environment and plans inside it. The World Models proposal trained a compact recurrent latent model and let a tiny controller learn entirely inside its imagined rollouts.^[198] PlaNet planned directly from pixels through a learned latent dynamics model,^[199] and the Dreamer line scaled this into a general agent that learns behaviours by imagining rollouts, with DreamerV3 mastering diverse domains under one fixed set of hyper-parameters.^[200] A different branch plans with a learned model of reward and value rather than pixels: MuZero reached superhuman play in Go, chess, shogi, and Atari without being told the rules,^[201] and EfficientZero brought that approach to human-level Atari from roughly two hours of experience.^[202]

Multi-agent architectures

When several agents share an environment, each one's learning shifts the others' world, so specialised architectures stabilise the joint problem. MADDPG gave each agent a centralised critic that sees every agent's actions while it still acts on local observations.^[203] For cooperative teams with a single shared reward, value-decomposition methods learn how to credit individuals: VDN summed per-agent values,^[204] QMIX generalised this to a monotonic mixing network,^[205] and COMA used a counterfactual baseline to isolate each agent's contribution.^[206] More recently, MAPPO showed that PPO paired with a centralised critic is a surprisingly strong baseline for these cooperative games.^[207]

Applications

• Game playing, from Atari and Go to real-time strategy.
• Robotics, control, and operations research.
• Aligning language models through human feedback.

Strengths and limitations