Neural Networks Taxonomy - 27. Domain-Specific Architectures

Details
Written by: Babak
Category: Blog
Hits: 527

Article Index

Page 28 of 29

27. Domain-Specific Architectures

Some of the largest breakthroughs are architectures tailored to a single domain. AlphaFold, in Fig 29, predicts protein structure by turning a sequence and its evolutionary relatives into pair features, refining them with attention (the Evoformer), and reading out 3-D atom coordinates — an accuracy leap on a decades-old problem.

A protein sequence with its multiple-sequence alignment feeding an Evoformer attention stack, a structure module, and 3-D atom coordinates, with a recycling loop
Fig 29. AlphaFold turns a sequence and its alignment into 3-D structure, refining the prediction through repeated recycling.

Origins and rise

AlphaFold combined attention over residues and residue pairs with a structure module and iterative recycling.[340] In other domains, TabNet brought attention-based feature selection to tabular data,[341] and vision-language-action models such as RT-2 repurposed multimodal Transformers as robot policies.[342]

Tabular deep learning

TabNet brought sequential attention to tabular data, selecting features per example for both accuracy and interpretability[341] while NODE generalised gradient-boosted trees into an end-to-end differentiable ensemble. The FT-Transformer and SAINT lines show that Transformer-style attention over columns, with intersample attention, can rival or beat gradient boosting on mid-sized tables.

Code generation

Programming languages are sequences too. CodeBERT extended masked-language pre-training to the bimodal code-and-text setting, and CodeT5 cast code understanding and generation into a unified text-to-text framework.[343] AlphaCode scaled the same Transformer backbone with massive search and filtering to solve unseen competitive-programming problems, pointing toward systems that write software from a natural-language specification.

Protein and molecular structure

Perhaps the highest-impact domain-specific architecture is AlphaFold 2, whose Evoformer exchanges information across pair and multiple-sequence-alignment representations to predict 3D protein structure at experimental accuracy.[340] RoseTTAFold offered a three-track alternative, and the ESM and ProtBERT protein language models bring the masked-pretraining recipe from natural language to amino-acid sequences, learning structure and function without labels.

Robotics and control

Robotics policies increasingly use large Transformer backbones. RT-1 showed that a vision-language-action Transformer can map camera input and natural-language commands to robot actions, and RT-2 demonstrated that co-fine-tuning a VLM on web-scale vision-language data and robot trajectories yields emergent semantic generalisation on the robot.[342] SayCan grounds a language model's proposals in affordances learned from the robot itself, chaining skills into long-horizon plans.

Applications

  • Protein-structure and molecular modelling.
  • Tabular prediction in finance and healthcare.
  • Language-conditioned robotic control.

Strengths and limitations

Strengths Limitations
Superhuman accuracy within their niche. Narrow, with heavy domain engineering.
Encode domain priors directly. Need large, curated datasets.
Reuse attention and multimodal ideas. Expensive to train and adapt.