Neural Networks Taxonomy - 28. Combined and Hybrid Architectures

28. Combined & Hybrid Architectures

Finally, many production systems are hybrids that chain families end to end. A classic pattern, in Fig 30, feeds convolutional features into a recurrent model: a CNN reads the image or video frames, and an RNN turns the resulting feature sequence into text — the recipe behind OCR and video captioning.

Origins and rise

The CRNN combined convolution, recurrence, and a sequence loss for image-based text recognition.^[344] As models scaled, mixture-of-experts hybrids such as the Switch Transformer routed each token to a subset of expert sub-networks, growing capacity without a proportional rise in compute.^[57] Other common hybrids marry CNNs with Transformers, or attention with recurrent sequence-to-sequence models.

CNN + RNN hybrids

Marrying a convolutional feature extractor with a recurrent head was the dominant recipe for video and sequence tasks before Transformers. A CNN-LSTM feeds per-frame CNN features into an LSTM for video captioning or action recognition, and CRNN (a CNN+RNN+CTC pipeline) became the standard architecture for optical character recognition, reading text as a sequence of image columns.

CNN + Transformer hybrids

Vision models increasingly blend the locality and translation invariance of convolution with the global context of self-attention. ConViT initialised a Vision Transformer with convolutional soft inductive biases, LeViT traded classification accuracy for fast inference by interleaving conv and attention stages, and MobileViT fused the mobile-CNN block design with Transformer blocks for lightweight mobile vision.

RNN + Attention (sequence-to-sequence)

The original attention mechanism was itself a hybrid: Bahdanau attention added a learned alignment between an encoder RNN and a decoder RNN, letting the decoder look back at all source hidden states for each output word.^[345] Luong attention simplified and generalised the scoring, and this encoder-decoder-plus-attention recipe dominated machine translation until the fully-attentional Transformer replaced the recurrent backbone.

Mixture-of-experts and ensembles

Scaling width is expensive, so mixture-of-experts (MoE) models sparsely activate a few expert sub-networks per token via a learned gating function. The Switch Transformer routed to a single expert per token to scale to trillions of parameters,^[57] GLaM and DeepSpeed-MoE refined the routing and load-balancing, and Mixtral brought sparse MoE into the open-weight LLM mainstream. Ensembles such as the basis-mixing in N-BEATS take a different route to the same goal: combine many diverse models into one robust forecast.

Applications

• Optical character recognition and video captioning.
• Scalable large language models via sparse experts.
• Any task where one family's output is another's input.

Strengths and limitations

Key takeaways

• A neural network is a stack of simple weighted units trained by backpropagation; every family here is that idea plus structure suited to a kind of data.
• Feedforward networks are the universal baseline and the output head of almost everything else.
• Convolutional networks add weight sharing for images; residual connections made them very deep.
• Recurrent networks handle sequences with a running state, and gates gave them longer memory.
• The Transformer swapped recurrence for attention, unlocking parallel training and today's large models across every modality.
• Generative models reuse those same backbones to sample new data rather than label it.
• Beyond these backbones, specialised families adapt the same machinery to other data and goals — graphs, spikes, and multiple modalities — while paradigms such as reinforcement learning and architecture search change how networks are trained and designed.
• The long tail specialises further still — recommenders, point clouds, time series, physics-informed solvers, and domain-specific models like AlphaFold — and the most capable systems are often hybrids that chain several families end to end.

Resources

The sources below are numbered in order of first appearance in the text; each bracketed marker such as ^[1] links to its entry. Entries are the seminal papers and datasets behind each family; the diagrams in this article are original schematics drawn for it, not reproductions of any third-party figure.

• [1] Deep Learning — LeCun, Bengio & Hinton, Nature, 2015. nature.com
• [2] A Logical Calculus of the Ideas Immanent in Nervous Activity — McCulloch & Pitts, Bulletin of Mathematical Biophysics, 1943. springer.com
• [3] The Perceptron: A Probabilistic Model — Rosenblatt, Psychological Review, 1958. apa.org
• [4] Learning Representations by Back-propagating Errors — Rumelhart, Hinton & Williams, Nature, 1986. nature.com
• [5] Multilayer Feedforward Networks are Universal Approximators — Hornik, Stinchcombe & White, Neural Networks, 1989. doi.org
• [6] Reducing the Dimensionality of Data with Neural Networks — Hinton & Salakhutdinov, Science, 2006. doi.org
• [7] A Fast Learning Algorithm for Deep Belief Nets — Hinton, Osindero & Teh, Neural Computation, 2006. doi.org
• [8] Extreme Learning Machine: Theory and Applications — Huang, Zhu & Siew, Neurocomputing, 2006. doi.org
• [9] Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position — Fukushima, Biological Cybernetics, 1980. doi.org
• [10] Gradient-Based Learning Applied to Document Recognition (LeNet-5) — LeCun, Bottou, Bengio & Haffner, Proc. IEEE, 1998. lecun.com
• [11] ImageNet Classification with Deep Convolutional Neural Networks (AlexNet) — Krizhevsky, Sutskever & Hinton, NeurIPS, 2012. papers.nips.cc
• [12] ImageNet: A Large-Scale Hierarchical Image Database — Deng et al., CVPR, 2009. Dataset, custom research license. image-net.org
• [13] Very Deep Convolutional Networks for Large-Scale Image Recognition (VGG) — Simonyan & Zisserman, ICLR, 2015. arxiv.org/abs/1409.1556
• [14] Deep Residual Learning for Image Recognition (ResNet) — He, Zhang, Ren & Sun, CVPR, 2016. arxiv.org/abs/1512.03385
• [15] Going Deeper with Convolutions (GoogLeNet / Inception) — Szegedy et al., CVPR, 2015. arxiv.org/abs/1409.4842
• [16] Densely Connected Convolutional Networks (DenseNet) — Huang, Liu, van der Maaten & Weinberger, CVPR, 2017. arxiv.org/abs/1608.06993
• [17] MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications — Howard et al., 2017. arxiv.org/abs/1704.04861
• [18] EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks — Tan & Le, ICML, 2019. arxiv.org/abs/1905.11946
• [19] A ConvNet for the 2020s (ConvNeXt) — Liu et al., CVPR, 2022. arxiv.org/abs/2201.03545
• [20] Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks — Ren, He, Girshick & Sun, NeurIPS, 2015. arxiv.org/abs/1506.01497
• [21] You Only Look Once: Unified, Real-Time Object Detection (YOLO) — Redmon, Divvala, Girshick & Farhadi, CVPR, 2016. arxiv.org/abs/1506.02640
• [22] Focal Loss for Dense Object Detection (RetinaNet) — Lin et al., ICCV, 2017. arxiv.org/abs/1708.02002
• [23] End-to-End Object Detection with Transformers (DETR) — Carion et al., ECCV, 2020. arxiv.org/abs/2005.12872
• [24] Fully Convolutional Networks for Semantic Segmentation — Long, Shelhamer & Darrell, CVPR, 2015. arxiv.org/abs/1411.4038
• [25] U-Net: Convolutional Networks for Biomedical Image Segmentation — Ronneberger, Fischer & Brox, MICCAI, 2015. arxiv.org/abs/1505.04597
• [26] Rethinking Atrous Convolution for Semantic Image Segmentation (DeepLabv3) — Chen, Papandreou, Schroff & Adam, 2017. arxiv.org/abs/1706.05587
• [27] Mask R-CNN — He, Gkioxari, Dollár & Girshick, ICCV, 2017. arxiv.org/abs/1703.06870
• [28] Segment Anything (SAM) — Kirillov et al., ICCV, 2023. arxiv.org/abs/2304.02643
• [29] Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset (I3D) — Carreira & Zisserman, CVPR, 2017. arxiv.org/abs/1705.07750
• [30] SlowFast Networks for Video Recognition — Feichtenhofer, Fan, Malik & He, ICCV, 2019. arxiv.org/abs/1812.03982
• [31] Image Super-Resolution Using Deep Convolutional Networks (SRCNN) — Dong, Loy, He & Tang, IEEE TPAMI, 2016. arxiv.org/abs/1501.00092
• [32] ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks — Wang et al., ECCV Workshops, 2018. arxiv.org/abs/1809.00219
• [33] SwinIR: Image Restoration Using Swin Transformer — Liang et al., ICCV Workshops, 2021. arxiv.org/abs/2108.10257
• [34] Finding Structure in Time — Elman, Cognitive Science, 1990. doi.org
• [35] Long Short-Term Memory (LSTM) — Hochreiter & Schmidhuber, Neural Computation, 1997. doi.org
• [36] Learning Phrase Representations using RNN Encoder-Decoder (GRU) — Cho et al., EMNLP, 2014. arxiv.org/abs/1406.1078
• [37] Bidirectional Recurrent Neural Networks — Schuster & Paliwal, IEEE Transactions on Signal Processing, 1997. doi.org
• [38] Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting — Shi et al., NeurIPS, 2015. arxiv.org/abs/1506.04214
• [39] Quasi-Recurrent Neural Networks — Bradbury, Merity, Xiong & Socher, ICLR, 2017. arxiv.org/abs/1611.01576
• [40] Simple Recurrent Units for Highly Parallelizable Recurrence — Lei et al., EMNLP, 2018. arxiv.org/abs/1709.02755
• [41] Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN — Li et al., CVPR, 2018. arxiv.org/abs/1803.04831
• [42] Neural Turing Machines — Graves, Wayne & Danihelka, 2014. arxiv.org/abs/1410.5401
• [43] Hybrid Computing Using a Neural Network with Dynamic External Memory (DNC) — Graves et al., Nature, 2016. nature.com
• [44] End-To-End Memory Networks — Sukhbaatar, Szlam, Weston & Fergus, NeurIPS, 2015. arxiv.org/abs/1503.08895
• [45] Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication (Echo State Network) — Jaeger & Haas, Science, 2004. doi.org
• [46] Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations (Liquid State Machine) — Maass, Natschläger & Markram, Neural Computation, 2002. doi.org
• [47] Attention Is All You Need — Vaswani et al., NeurIPS, 2017. arxiv.org/abs/1706.03762
• [48] BERT: Pre-training of Deep Bidirectional Transformers — Devlin et al., NAACL, 2019. arxiv.org/abs/1810.04805
• [49] Language Models are Few-Shot Learners (GPT-3) — Brown et al., NeurIPS, 2020. arxiv.org/abs/2005.14165
• [50] An Image is Worth 16×16 Words (Vision Transformer) — Dosovitskiy et al., ICLR, 2021. arxiv.org/abs/2010.11929
• [51] RoBERTa: A Robustly Optimized BERT Pretraining Approach — Liu et al., 2019. arxiv.org/abs/1907.11692
• [52] ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators — Clark et al., ICLR, 2020. arxiv.org/abs/2003.10555
• [53] DeBERTa: Decoding-enhanced BERT with Disentangled Attention — He et al., ICLR, 2021. arxiv.org/abs/2006.03654
• [54] Training Compute-Optimal Large Language Models (Chinchilla) — Hoffmann et al., NeurIPS, 2022. arxiv.org/abs/2203.15556
• [55] PaLM: Scaling Language Modeling with Pathways — Chowdhery et al., JMLR, 2023. arxiv.org/abs/2204.02311
• [56] LLaMA: Open and Efficient Foundation Language Models — Touvron et al., 2023. arxiv.org/abs/2302.13971
• [57] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — Fedus, Zoph & Shazeer, JMLR, 2022. arxiv.org/abs/2101.03961
• [58] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5) — Raffel et al., JMLR, 2020. arxiv.org/abs/1910.10683
• [59] BART: Denoising Sequence-to-Sequence Pre-training — Lewis et al., ACL, 2020. arxiv.org/abs/1910.13461
• [60] Training Data-Efficient Image Transformers & Distillation through Attention (DeiT) — Touvron et al., ICML, 2021. arxiv.org/abs/2012.12877
• [61] Swin Transformer: Hierarchical Vision Transformer using Shifted Windows — Liu et al., ICCV, 2021. arxiv.org/abs/2103.14030
• [62] BEiT: BERT Pre-Training of Image Transformers — Bao et al., ICLR, 2022. arxiv.org/abs/2106.08254
• [63] Masked Autoencoders Are Scalable Vision Learners — He et al., CVPR, 2022. arxiv.org/abs/2111.06377
• [64] Emerging Properties in Self-Supervised Vision Transformers (DINO) — Caron et al., ICCV, 2021. arxiv.org/abs/2104.14294
• [65] Longformer: The Long-Document Transformer — Beltagy et al., 2020. arxiv.org/abs/2004.05150
• [66] Big Bird: Transformers for Longer Sequences — Zaheer et al., NeurIPS, 2020. arxiv.org/abs/2007.14062
• [67] Reformer: The Efficient Transformer — Kitaev et al., ICLR, 2020. arxiv.org/abs/2001.04451
• [68] Rethinking Attention with Performers — Choromanski et al., ICLR, 2021. arxiv.org/abs/2009.14794
• [69] FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — Dao et al., NeurIPS, 2022. arxiv.org/abs/2205.14135
• [70] Decision Transformer: Reinforcement Learning via Sequence Modeling — Chen et al., NeurIPS, 2021. arxiv.org/abs/2106.01345
• [71] A Generalist Agent (Gato) — Reed et al., TMLR, 2022. arxiv.org/abs/2205.06175
• [72] Generative Adversarial Nets (GAN) — Goodfellow et al., NeurIPS, 2014. arxiv.org/abs/1406.2661
• [73] Auto-Encoding Variational Bayes (VAE) — Kingma & Welling, ICLR, 2014. arxiv.org/abs/1312.6114
• [74] Denoising Diffusion Probabilistic Models (DDPM) — Ho, Jain & Abbeel, NeurIPS, 2020. arxiv.org/abs/2006.11239
• [75] High-Resolution Image Synthesis with Latent Diffusion Models (Stable Diffusion) — Rombach et al., CVPR, 2022. arxiv.org/abs/2112.10752
• [76] Unsupervised Representation Learning with Deep Convolutional GANs (DCGAN) — Radford, Metz & Chintala, ICLR, 2016. arxiv.org/abs/1511.06434
• [77] Wasserstein GAN — Arjovsky, Chintala & Bottou, ICML, 2017. arxiv.org/abs/1701.07875
• [78] Self-Attention Generative Adversarial Networks (SAGAN) — Zhang et al., ICML, 2019. arxiv.org/abs/1805.08318
• [79] Progressive Growing of GANs for Improved Quality, Stability, and Variation — Karras et al., ICLR, 2018. arxiv.org/abs/1710.10196
• [80] A Style-Based Generator Architecture for GANs (StyleGAN) — Karras, Laine & Aila, CVPR, 2019. arxiv.org/abs/1812.04948
• [81] Large Scale GAN Training for High Fidelity Natural Image Synthesis (BigGAN) — Brock, Donahue & Simonyan, ICLR, 2019. arxiv.org/abs/1809.11096
• [82] Image-to-Image Translation with Conditional Adversarial Networks (Pix2Pix) — Isola et al., CVPR, 2017. arxiv.org/abs/1611.07004
• [83] Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks (CycleGAN) — Zhu et al., ICCV, 2017. arxiv.org/abs/1703.10593
• [84] Neural Discrete Representation Learning (VQ-VAE) — van den Oord, Vinyals & Kavukcuoglu, NeurIPS, 2017. arxiv.org/abs/1711.00937
• [85] Taming Transformers for High-Resolution Image Synthesis (VQGAN) — Esser, Rombach & Ommer, CVPR, 2021. arxiv.org/abs/2012.09841
• [86] Pixel Recurrent Neural Networks — van den Oord, Kalchbrenner & Kavukcuoglu, ICML, 2016. arxiv.org/abs/1601.06759
• [87] WaveNet: A Generative Model for Raw Audio — van den Oord et al., 2016. arxiv.org/abs/1609.03499
• [88] Denoising Diffusion Implicit Models (DDIM) — Song, Meng & Ermon, ICLR, 2021. arxiv.org/abs/2010.02502
• [89] Score-Based Generative Modeling through Stochastic Differential Equations — Song et al., ICLR, 2021. arxiv.org/abs/2011.13456
• [90] Diffusion Models Beat GANs on Image Synthesis — Dhariwal & Nichol, NeurIPS, 2021. arxiv.org/abs/2105.05233
• [91] SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis — Podell et al., 2023. arxiv.org/abs/2307.01952
• [92] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling — Lu et al., NeurIPS, 2022. arxiv.org/abs/2206.00927
• [93] Density Estimation using Real NVP — Dinh, Sohl-Dickstein & Bengio, ICLR, 2017. arxiv.org/abs/1605.08803
• [94] Glow: Generative Flow with Invertible 1×1 Convolutions — Kingma & Dhariwal, NeurIPS, 2018. arxiv.org/abs/1807.03039
• [95] Your Classifier is Secretly an Energy Based Model (JEM) — Grathwohl et al., ICLR, 2020. arxiv.org/abs/1912.03263
• [96] Semi-Supervised Classification with Graph Convolutional Networks (GCN) — Kipf & Welling, ICLR, 2017. arxiv.org/abs/1609.02907
• [97] Graph Attention Networks (GAT) — Veličković et al., ICLR, 2018. arxiv.org/abs/1710.10903
• [98] Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering (ChebNet) — Defferrard, Bresson & Vandergheynst, NeurIPS, 2016. arxiv.org/abs/1606.09375
• [99] Inductive Representation Learning on Large Graphs (GraphSAGE) — Hamilton, Ying & Leskovec, NeurIPS, 2017. arxiv.org/abs/1706.02216
• [100] Neural Message Passing for Quantum Chemistry (MPNN) — Gilmer et al., ICML, 2017. arxiv.org/abs/1704.01212
• [101] How Powerful are Graph Neural Networks? (GIN) — Xu et al., ICLR, 2019. arxiv.org/abs/1810.00826
• [102] SchNet: A Continuous-Filter Convolutional Neural Network for Modeling Quantum Interactions — Schütt et al., NeurIPS, 2017. arxiv.org/abs/1706.08566
• [103] SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks — Fuchs et al., NeurIPS, 2020. arxiv.org/abs/2006.10503
• [104] Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition (ST-GCN) — Yan, Xiong & Lin, AAAI, 2018. arxiv.org/abs/1801.07455
• [105] Graph WaveNet for Deep Spatial-Temporal Graph Modeling — Wu et al., IJCAI, 2019. arxiv.org/abs/1906.00121
• [106] Do Transformers Really Perform Bad for Graph Representation? (Graphormer) — Ying et al., NeurIPS, 2021. arxiv.org/abs/2106.05234
• [107] Graph Contrastive Learning with Augmentations (GraphCL) — You et al., NeurIPS, 2020. arxiv.org/abs/2010.13902
• [108] Self-Organized Formation of Topologically Correct Feature Maps — Kohonen, Biological Cybernetics, 1982. doi.org
• [109] The Growing Hierarchical Self-Organizing Map: Exploratory Analysis of High-Dimensional Data — Rauber, Merkl & Dittenbach, IEEE Transactions on Neural Networks, 2002. doi.org
• [110] ‘Neural-Gas’ Network for Vector Quantization and its Application to Time-Series Prediction — Martinetz, Berkovich & Schulten, IEEE Transactions on Neural Networks, 1993. doi.org
• [111] A Self-Organising Network that Grows When Required (GWR) — Marsland, Shapiro & Nehmzow, Neural Networks, 2002. doi.org
• [112] A Review of Learning Vector Quantization Classifiers — Nova & Estévez, Neural Computing and Applications, 2014. doi.org
• [113] Dynamic Routing Between Capsules — Sabour, Frosst & Hinton, NeurIPS, 2017. arxiv.org/abs/1710.09829
• [114] Matrix Capsules with EM Routing — Hinton, Sabour & Frosst, ICLR, 2018. openreview.net
• [115] DeepCaps: Going Deeper with Capsule Networks — Rajasegaran et al., CVPR, 2019. arxiv.org/abs/1904.09546
• [116] Stacked Capsule Autoencoders — Kosiorek, Sabour, Teh & Hinton, NeurIPS, 2019. arxiv.org/abs/1906.06818
• [117] 3D Point Capsule Networks — Zhao, Birdal, Deng & Tombari, CVPR, 2019. arxiv.org/abs/1812.10775
• [118] Signature Verification using a Siamese Time Delay Neural Network — Bromley et al., NeurIPS, 1993. papers.nips.cc
• [119] FaceNet: A Unified Embedding for Face Recognition and Clustering — Schroff, Kalenichenko & Philbin, CVPR, 2015. arxiv.org/abs/1503.03832
• [120] Dimensionality Reduction by Learning an Invariant Mapping (Contrastive Loss) — Hadsell, Chopra & LeCun, CVPR, 2006. doi.org
• [121] Deep Metric Learning via Lifted Structured Feature Embedding — Song, Xiang, Jegelka & Savarese, CVPR, 2016. arxiv.org/abs/1511.06452
• [122] Matching Networks for One Shot Learning — Vinyals et al., NeurIPS, 2016. arxiv.org/abs/1606.04080
• [123] Prototypical Networks for Few-shot Learning — Snell, Swersky & Zemel, NeurIPS, 2017. arxiv.org/abs/1703.05175
• [124] Learning to Compare: Relation Network for Few-Shot Learning — Sung et al., CVPR, 2018. arxiv.org/abs/1711.06025
• [125] A Simple Framework for Contrastive Learning of Visual Representations (SimCLR) — Chen, Kornblith, Norouzi & Hinton, ICML, 2020. arxiv.org/abs/2002.05709
• [126] Momentum Contrast for Unsupervised Visual Representation Learning (MoCo) — He, Fan, Wu, Xie & Girshick, CVPR, 2020. arxiv.org/abs/1911.05722
• [127] Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (BYOL) — Grill et al., NeurIPS, 2020. arxiv.org/abs/2006.07733
• [128] Exploring Simple Siamese Representation Learning (SimSiam) — Chen & He, CVPR, 2021. arxiv.org/abs/2011.10566
• [129] Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (SwAV) — Caron et al., NeurIPS, 2020. arxiv.org/abs/2006.09882
• [130] Barlow Twins: Self-Supervised Learning via Redundancy Reduction — Zbontar, Jing, Misra, LeCun & Deny, ICML, 2021. arxiv.org/abs/2103.03230
• [131] VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning — Bardes, Ponce & LeCun, ICLR, 2022. arxiv.org/abs/2105.04906
• [132] ArcFace: Additive Angular Margin Loss for Deep Face Recognition — Deng, Guo, Yang, Xue, Kotsia & Zafeiriou, CVPR, 2019. arxiv.org/abs/1801.07698
• [133] Proxy Anchor Loss for Deep Metric Learning — Kim, Kim, Cho & Kwak, CVPR, 2020. arxiv.org/abs/2003.13911
• [134] Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning — Wang, Han, Huang, Dong & Scott, CVPR, 2019. arxiv.org/abs/1904.06627
• [135] Circle Loss: A Unified Perspective of Pair Similarity Optimization — Sun et al., CVPR, 2020. arxiv.org/abs/2002.10857
• [136] Neural Ordinary Differential Equations — Chen, Rubanova, Bettencourt & Duvenaud, NeurIPS, 2018. arxiv.org/abs/1806.07366
• [137] Augmented Neural ODEs — Dupont, Doucet & Teh, NeurIPS, 2019. arxiv.org/abs/1904.01681
• [138] Latent ODEs for Irregularly-Sampled Time Series — Rubanova, Chen & Duvenaud, NeurIPS, 2019. arxiv.org/abs/1907.03907
• [139] Neural Controlled Differential Equations for Irregular Time Series — Kidger, Morrill, Foster & Lyons, NeurIPS, 2020. arxiv.org/abs/2005.08926
• [140] FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models — Grathwohl, Chen, Bettencourt, Sutskever & Duvenaud, ICLR, 2019. arxiv.org/abs/1810.01367
• [141] Scalable Gradients for Stochastic Differential Equations — Li, Wong, Chen & Duvenaud, AISTATS, 2020. arxiv.org/abs/2001.01328
• [142] Neural SDEs as Infinite-Dimensional GANs — Kidger, Foster, Li, Oberhauser & Lyons, ICML, 2021. arxiv.org/abs/2102.03657
• [143] Networks of Spiking Neurons: The Third Generation of Neural Network Models — Maass, Neural Networks, 1997. doi.org
• [144] Surrogate Gradient Learning in Spiking Neural Networks — Neftci, Mostafa & Zenke, IEEE Signal Processing Magazine, 2019. arxiv.org/abs/1901.09948
• [145] Simple Model of Spiking Neurons — Izhikevich, IEEE Transactions on Neural Networks, 2003. doi.org
• [146] Adaptive Exponential Integrate-and-Fire Model as an Effective Description of Neuronal Activity (AdEx) — Brette & Gerstner, Journal of Neurophysiology, 2005. doi.org
• [147] Unsupervised Learning of Digit Recognition Using Spike-Timing-Dependent Plasticity — Diehl & Cook, Frontiers in Computational Neuroscience, 2015. doi.org
• [148] Conversion of Continuous-Valued Deep Networks to Efficient Event-Driven Networks for Image Classification — Rueckauer et al., Frontiers in Neuroscience, 2017. doi.org
• [149] Going Deeper in Spiking Neural Networks: VGG and Residual Architectures — Sengupta et al., Frontiers in Neuroscience, 2019. arxiv.org/abs/1802.02627
• [150] SLAYER: Spike Layer Error Reassignment in Time — Shrestha & Orchard, NeurIPS, 2018. arxiv.org/abs/1810.08646
• [151] Going Deeper With Directly-Trained Larger Spiking Neural Networks — Zheng et al., AAAI, 2021. arxiv.org/abs/2011.05280
• [152] Spikformer: When Spiking Neural Network Meets Transformer — Zhou et al., ICLR, 2023. arxiv.org/abs/2209.15425
• [153] Long Short-Term Memory and Learning-to-Learn in Networks of Spiking Neurons (LSNN) — Bellec et al., NeurIPS, 2018. arxiv.org/abs/1803.09574
• [154] A Million Spiking-Neuron Integrated Circuit with a Scalable Communication Network and Interface (TrueNorth) — Merolla et al., Science, 2014. doi.org
• [155] Loihi: A Neuromorphic Manycore Processor with On-Chip Learning — Davies et al., IEEE Micro, 2018. doi.org
• [156] Squeeze-and-Excitation Networks (SENet) — Hu, Shen & Sun, CVPR, 2018. arxiv.org/abs/1709.01507
• [157] CBAM: Convolutional Block Attention Module — Woo et al., ECCV, 2018. arxiv.org/abs/1807.06521
• [158] ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks — Wang et al., CVPR, 2020. arxiv.org/abs/1910.03151
• [159] Selective Kernel Networks (SKNet) — Li et al., CVPR, 2019. arxiv.org/abs/1903.06586
• [160] BAM: Bottleneck Attention Module — Park et al., BMVC, 2018. arxiv.org/abs/1807.06514
• [161] Non-local Neural Networks — Wang, Girshick, Gupta & He, CVPR, 2018. arxiv.org/abs/1711.07971
• [162] GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond — Cao et al., ICCV Workshops, 2019. arxiv.org/abs/1904.11492
• [163] CCNet: Criss-Cross Attention for Semantic Segmentation — Huang et al., ICCV, 2019. arxiv.org/abs/1811.11721
• [164] Stand-Alone Self-Attention in Vision Models — Ramachandran et al., NeurIPS, 2019. arxiv.org/abs/1906.05909
• [165] Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation — Wang et al., ECCV, 2020. arxiv.org/abs/2003.07853
• [166] Neural Architecture Search with Reinforcement Learning — Zoph & Le, ICLR, 2017. arxiv.org/abs/1611.01578
• [167] DARTS: Differentiable Architecture Search — Liu, Simonyan & Yang, ICLR, 2019. arxiv.org/abs/1806.09055
• [168] Learning Transferable Architectures for Scalable Image Recognition (NASNet) — Zoph, Vasudevan, Shlens & Le, CVPR, 2018. arxiv.org/abs/1707.07012
• [169] Efficient Neural Architecture Search via Parameter Sharing (ENAS) — Pham et al., ICML, 2018. arxiv.org/abs/1802.03268
• [170] PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search — Xu et al., ICLR, 2020. arxiv.org/abs/1907.05737
• [171] Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation (P-DARTS) — Chen et al., ICCV, 2019. arxiv.org/abs/1904.12760
• [172] Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search — Chu et al., ECCV, 2020. arxiv.org/abs/1911.12126
• [173] DARTS-: Robustly Stepping out of Performance Collapse Without Indicators — Chu et al., ICLR, 2021. arxiv.org/abs/2009.01027
• [174] MnasNet: Platform-Aware Neural Architecture Search for Mobile — Tan et al., CVPR, 2019. arxiv.org/abs/1807.11626
• [175] MixConv: Mixed Depthwise Convolutional Kernels (MixNet) — Tan & Le, BMVC, 2019. arxiv.org/abs/1907.09595
• [176] ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware — Cai, Zhu & Han, ICLR, 2019. arxiv.org/abs/1812.00332
• [177] FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search — Wu et al., CVPR, 2019. arxiv.org/abs/1812.03443
• [178] HAT: Hardware-Aware Transformers for Efficient Natural Language Processing — Wang et al., ACL, 2020. arxiv.org/abs/2005.14187
• [179] Once-for-All: Train One Network and Specialize it for Efficient Deployment — Cai et al., ICLR, 2020. arxiv.org/abs/1908.09791
• [180] BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models — Yu et al., ECCV, 2020. arxiv.org/abs/2003.11142
• [181] SCARLET-NAS: Bridging the Gap between Stability and Scalability in Weight-sharing Neural Architecture Search — Chu et al., 2021. arxiv.org/abs/1908.06022
• [182] AutoFormer: Searching Transformers for Visual Recognition — Chen et al., ICCV, 2021. arxiv.org/abs/2107.00651
• [183] Designing Network Design Spaces (RegNet) — Radosavovic et al., CVPR, 2020. arxiv.org/abs/2003.13678
• [184] Human-Level Control through Deep Reinforcement Learning (DQN) — Mnih et al., Nature, 2015. nature.com
• [185] Proximal Policy Optimization Algorithms (PPO) — Schulman et al., 2017. arxiv.org/abs/1707.06347
• [186] Deep Reinforcement Learning with Double Q-learning (Double DQN) — van Hasselt, Guez & Silver, AAAI, 2016. arxiv.org/abs/1509.06461
• [187] Dueling Network Architectures for Deep Reinforcement Learning — Wang et al., ICML, 2016. arxiv.org/abs/1511.06581
• [188] A Distributional Perspective on Reinforcement Learning (C51) — Bellemare, Dabney & Munos, ICML, 2017. arxiv.org/abs/1707.06887
• [189] Distributional Reinforcement Learning with Quantile Regression (QR-DQN) — Dabney et al., AAAI, 2018. arxiv.org/abs/1710.10044
• [190] Implicit Quantile Networks for Distributional Reinforcement Learning (IQN) — Dabney et al., ICML, 2018. arxiv.org/abs/1806.06923
• [191] Rainbow: Combining Improvements in Deep Reinforcement Learning — Hessel et al., AAAI, 2018. arxiv.org/abs/1710.02298
• [192] Distributed Prioritized Experience Replay (Ape-X) — Horgan et al., ICLR, 2018. arxiv.org/abs/1803.00933
• [193] Asynchronous Methods for Deep Reinforcement Learning (A3C) — Mnih et al., ICML, 2016. arxiv.org/abs/1602.01783
• [194] Trust Region Policy Optimization (TRPO) — Schulman et al., ICML, 2015. arxiv.org/abs/1502.05477
• [195] Continuous Control with Deep Reinforcement Learning (DDPG) — Lillicrap et al., ICLR, 2016. arxiv.org/abs/1509.02971
• [196] Addressing Function Approximation Error in Actor-Critic Methods (TD3) — Fujimoto, van Hoof & Meger, ICML, 2018. arxiv.org/abs/1802.09477
• [197] Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning (SAC) — Haarnoja et al., ICML, 2018. arxiv.org/abs/1801.01290
• [198] World Models — Ha & Schmidhuber, NeurIPS, 2018. arxiv.org/abs/1803.10122
• [199] Learning Latent Dynamics for Planning from Pixels (PlaNet) — Hafner et al., ICML, 2019. arxiv.org/abs/1811.04551
• [200] Mastering Diverse Domains through World Models (DreamerV3) — Hafner et al., 2023. arxiv.org/abs/2301.04104
• [201] Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model (MuZero) — Schrittwieser et al., Nature, 2020. arxiv.org/abs/1911.08265
• [202] Mastering Atari Games with Limited Data (EfficientZero) — Ye et al., NeurIPS, 2021. arxiv.org/abs/2111.00210
• [203] Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments (MADDPG) — Lowe et al., NeurIPS, 2017. arxiv.org/abs/1706.02275
• [204] Value-Decomposition Networks for Cooperative Multi-Agent Learning (VDN) — Sunehag et al., AAMAS, 2018. arxiv.org/abs/1706.05296
• [205] QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning — Rashid et al., ICML, 2018. arxiv.org/abs/1803.11485
• [206] Counterfactual Multi-Agent Policy Gradients (COMA) — Foerster et al., AAAI, 2018. arxiv.org/abs/1705.08926
• [207] The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games (MAPPO) — Yu et al., NeurIPS, 2022. arxiv.org/abs/2103.01955
• [208] Learning Transferable Visual Models From Natural Language Supervision (CLIP) — Radford et al., ICML, 2021. arxiv.org/abs/2103.00020
• [209] Flamingo: a Visual Language Model for Few-Shot Learning — Alayrac et al., NeurIPS, 2022. arxiv.org/abs/2204.14198
• [210] Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (ALIGN) — Jia et al., ICML, 2021. arxiv.org/abs/2102.05918
• [211] LiT: Zero-Shot Transfer with Locked-image Text Tuning — Zhai et al., CVPR, 2022. arxiv.org/abs/2111.07991
• [212] Sigmoid Loss for Language Image Pre-Training (SigLIP) — Zhai et al., ICCV, 2023. arxiv.org/abs/2303.15343
• [213] Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering — Anderson et al., CVPR, 2018. arxiv.org/abs/1707.07998
• [214] ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks — Lu et al., NeurIPS, 2019. arxiv.org/abs/1908.02265
• [215] LXMERT: Learning Cross-Modality Encoder Representations from Transformers — Tan & Bansal, EMNLP, 2019. arxiv.org/abs/1908.07490
• [216] UNITER: UNiversal Image-TExt Representation Learning — Chen et al., ECCV, 2020. arxiv.org/abs/1909.11740
• [217] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation — Li et al., ICML, 2022. arxiv.org/abs/2201.12086
• [218] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models — Li et al., ICML, 2023. arxiv.org/abs/2301.12597
• [219] PaLI: A Jointly-Scaled Multilingual Language-Image Model — Chen et al., ICLR, 2023. arxiv.org/abs/2209.06794
• [220] MDETR: Modulated Detection for End-to-End Multi-Modal Understanding — Kamath et al., ICCV, 2021. arxiv.org/abs/2104.12763
• [221] Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection — Liu et al., ECCV, 2024. arxiv.org/abs/2303.05499
• [222] Kosmos-2: Grounding Multimodal Large Language Models to the World — Peng et al., ICLR, 2024. arxiv.org/abs/2306.14824
• [223] LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking — Huang et al., ACM MM, 2022. arxiv.org/abs/2204.08387
• [224] OCR-free Document Understanding Transformer (Donut) — Kim et al., ECCV, 2022. arxiv.org/abs/2111.15664
• [225] Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding — Lee et al., ICML, 2023. arxiv.org/abs/2210.03347
• [226] CoCa: Contrastive Captioners are Image-Text Foundation Models — Yu et al., TMLR, 2022. arxiv.org/abs/2205.01917
• [227] Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks — Lu et al., ICLR, 2023. arxiv.org/abs/2206.08916
• [228] Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks (BEiT-3) — Wang et al., CVPR, 2023. arxiv.org/abs/2208.10442
• [229] ImageBind: One Embedding Space To Bind Them All — Girdhar et al., CVPR, 2023. arxiv.org/abs/2305.05665
• [230] Wide & Deep Learning for Recommender Systems — Cheng et al., DLRS, 2016. arxiv.org/abs/1606.07792
• [231] Neural Collaborative Filtering (NCF) — He et al., WWW, 2017. arxiv.org/abs/1708.05031
• [232] Deep Learning Recommendation Model (DLRM) — Naumov et al., 2019. arxiv.org/abs/1906.00091
• [233] DeepFM: A Factorization-Machine based Neural Network for CTR Prediction — Guo et al., IJCAI, 2017. arxiv.org/abs/1703.04247
• [234] Deep & Cross Network for Ad Click Predictions — Wang et al., ADKDD, 2017. arxiv.org/abs/1708.05123
• [235] DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems — Wang et al., WWW, 2021. arxiv.org/abs/2008.13535
• [236] xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems — Lian et al., KDD, 2018. arxiv.org/abs/1803.05170
• [237] AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks — Song et al., CIKM, 2019. arxiv.org/abs/1810.11921
• [238] Session-based Recommendations with Recurrent Neural Networks (GRU4Rec) — Hidasi et al., ICLR, 2016. arxiv.org/abs/1511.06939
• [239] Self-Attentive Sequential Recommendation (SASRec) — Kang & McAuley, ICDM, 2018. arxiv.org/abs/1808.09781
• [240] BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer — Sun et al., CIKM, 2019. arxiv.org/abs/1904.06690
• [241] Session-based Recommendation with Graph Neural Networks (SR-GNN) — Wu et al., AAAI, 2019. arxiv.org/abs/1811.00855
• [242] Learning Deep Structured Semantic Models for Web Search using Clickthrough Data (DSSM) — Huang et al., CIKM, 2013. microsoft.com/en-us/research/publication
• [243] Deep Neural Networks for YouTube Recommendations — Covington, Adams & Sargin, RecSys, 2016. dl.acm.org/doi/10.1145/2959100.2959190
• [244] PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation — Qi, Su, Mo & Guibas, CVPR, 2017. arxiv.org/abs/1612.00593
• [245] PointNet++: Deep Hierarchical Feature Learning on Point Sets — Qi, Yi, Su & Guibas, NeurIPS, 2017. arxiv.org/abs/1706.02413
• [246] NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis — Mildenhall et al., ECCV, 2020. arxiv.org/abs/2003.08934
• [247] PointCNN: Convolution On X-Transformed Points — Li et al., NeurIPS, 2018. arxiv.org/abs/1801.07791
• [248] PointConv: Deep Convolutional Networks on 3D Point Clouds — Wu, Qi & Fuxin, CVPR, 2019. arxiv.org/abs/1811.07246
• [249] KPConv: Flexible and Deformable Convolution for Point Clouds — Thomas et al., ICCV, 2019. arxiv.org/abs/1904.08889
• [250] SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters — Xu et al., ECCV, 2018. arxiv.org/abs/1803.11527
• [251] PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds — Xu et al., CVPR, 2021. arxiv.org/abs/2103.14635
• [252] Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework (PointMLP) — Ma et al., ICLR, 2022. arxiv.org/abs/2202.07123
• [253] Dynamic Graph CNN for Learning on Point Clouds — Wang et al., ACM Transactions on Graphics, 2019. arxiv.org/abs/1801.07829
• [254] PCT: Point Cloud Transformer — Guo et al., Computational Visual Media, 2021. arxiv.org/abs/2012.09688
• [255] Point Transformer — Zhao et al., ICCV, 2021. arxiv.org/abs/2012.09164
• [256] Stratified Transformer for 3D Point Cloud Segmentation — Lai et al., CVPR, 2022. arxiv.org/abs/2203.14508
• [257] OctFormer: Octree-based Transformers for 3D Point Clouds — Wang, ACM Transactions on Graphics, 2023. arxiv.org/abs/2305.03045
• [258] VoxNet: A 3D Convolutional Neural Network for Real-Time Object Recognition — Maturana & Scherer, IROS, 2015. doi.org/10.1109/IROS.2015.7353481
• [259] 3D Semantic Segmentation with Submanifold Sparse Convolutional Networks — Graham, Engelcke & van der Maaten, CVPR, 2018. arxiv.org/abs/1711.10275
• [260] 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks — Choy, Gwak & Savarese, CVPR, 2019. arxiv.org/abs/1904.08755
• [261] SECOND: Sparsely Embedded Convolutional Detection — Yan, Mao & Li, Sensors, 2018. doi.org/10.3390/s18103337
• [262] DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation — Park et al., CVPR, 2019. arxiv.org/abs/1901.05103
• [263] Occupancy Networks: Learning 3D Reconstruction in Function Space — Mescheder et al., CVPR, 2019. arxiv.org/abs/1812.03828
• [264] Convolutional Occupancy Networks — Peng et al., ECCV, 2020. arxiv.org/abs/2003.04618
• [265] PointNetLK: Robust & Efficient Point Cloud Registration using PointNet — Aoki et al., CVPR, 2019. arxiv.org/abs/1903.05711
• [266] Deep Closest Point: Learning Representations for Point Cloud Registration — Wang & Solomon, ICCV, 2019. arxiv.org/abs/1905.03304
• [267] RPM-Net: Robust Point Matching using Learned Features — Yew & Lee, CVPR, 2020. arxiv.org/abs/2003.13479
• [268] PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency — Bai et al., CVPR, 2021. arxiv.org/abs/2103.05465
• [269] An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (TCN) — Bai, Kolter & Koltun, 2018. arxiv.org/abs/1803.01271
• [270] N-BEATS: Neural Basis Expansion Analysis for Time Series Forecasting — Oreshkin et al., ICLR, 2020. arxiv.org/abs/1905.10437
• [271] Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting — Zhou et al., AAAI, 2021. arxiv.org/abs/2012.07436
• [272] SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction — Liu et al., NeurIPS, 2022. arxiv.org/abs/2106.09305
• [273] N-HiTS: Neural Hierarchical Interpolation for Time Series Forecasting — Challu et al., AAAI, 2023. arxiv.org/abs/2201.12886
• [274] DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks — Salinas, Flunkert & Gasthaus, Int. J. Forecasting, 2020. arxiv.org/abs/1704.04110
• [275] A Multi-Horizon Quantile Recurrent Forecaster — Wen et al., NeurIPS Time Series Workshop, 2017. arxiv.org/abs/1711.11053
• [276] Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting — Lim et al., Int. J. Forecasting, 2021. arxiv.org/abs/1912.09363
• [277] Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting — Wu et al., NeurIPS, 2021. arxiv.org/abs/2106.13008
• [278] FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting — Zhou et al., ICML, 2022. arxiv.org/abs/2201.12740
• [279] A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (PatchTST) — Nie et al., ICLR, 2023. arxiv.org/abs/2211.14730
• [280] TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis — Wu et al., ICLR, 2023. arxiv.org/abs/2210.02186
• [281] USAD: UnSupervised Anomaly Detection on Multivariate Time Series — Audibert et al., KDD, 2020. doi.org/10.1145/3394486.3403392
• [282] TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Data — Tuli, Casale & Jennings, VLDB, 2022. arxiv.org/abs/2201.07284
• [283] Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy — Xu et al., ICLR, 2022. arxiv.org/abs/2110.02642
• [284] Deep Sets — Zaheer et al., NeurIPS, 2017. arxiv.org/abs/1703.06114
• [285] Set Transformer — Lee et al., ICML, 2019. arxiv.org/abs/1810.00825
• [286] Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs — Murphy et al., ICLR, 2019. arxiv.org/abs/1811.01900
• [287] Rep the Set: Neural Networks for Learning Set Representations — Skianis et al., AISTATS, 2020. arxiv.org/abs/1904.01962
• [288] On the Limitations of Representing Functions on Sets — Wagstaff et al., ICML, 2019. arxiv.org/abs/1901.09006
• [289] Deep Set Prediction Networks — Zhang, Hare & Prügel-Bennett, NeurIPS, 2019. arxiv.org/abs/1906.06565
• [290] Object-Centric Learning with Slot Attention — Locatello et al., NeurIPS, 2020. arxiv.org/abs/2006.15055
• [291] Deep Equilibrium Models (DEQ) — Bai, Kolter & Koltun, NeurIPS, 2019. arxiv.org/abs/1909.01377
• [292] Multiscale Deep Equilibrium Models (MDEQ) — Bai, Koltun & Kolter, NeurIPS, 2020. arxiv.org/abs/2006.08656
• [293] Implicit Deep Learning — El Ghaoui et al., SIAM J. Mathematics of Data Science, 2021. arxiv.org/abs/1908.06315
• [294] Monotone Operator Equilibrium Networks — Winston & Kolter, NeurIPS, 2020. arxiv.org/abs/2006.08591
• [295] Stabilizing Equilibrium Models by Jacobian Regularization — Bai, Koltun & Kolter, ICML, 2021. arxiv.org/abs/2106.14342
• [296] Implicit Graph Neural Networks — Gu et al., NeurIPS, 2020. arxiv.org/abs/2009.06211
• [297] Deep Equilibrium Optical Flow Estimation — Bai et al., CVPR, 2022. arxiv.org/abs/2204.08442
• [298] Evolving Neural Networks through Augmenting Topologies (NEAT) — Stanley & Miikkulainen, Evolutionary Computation, 2002. doi.org
• [299] A Hypercube-Based Encoding for Evolving Large-Scale Neural Networks (HyperNEAT) — Stanley, D’Ambrosio & Gauci, Artificial Life, 2009. doi.org
• [300] Evolution Strategies as a Scalable Alternative to Reinforcement Learning — Salimans et al., 2017. arxiv.org/abs/1703.03864
• [301] Evolving Deep Neural Networks (CoDeepNEAT) — Miikkulainen et al., 2017. arxiv.org/abs/1703.00548
• [302] An Enhanced Hypercube-Based Encoding for Evolving the Placement, Density and Connectivity of Neurons (ES-HyperNEAT) — Risi & Stanley, Artificial Life, 2012. doi.org/10.1162/artl_a_00071
• [303] Evolutionary Reinforcement Learning of Artificial Neural Networks (EANT2) — Siebel & Sommer, Int. J. Hybrid Intelligent Systems, 2007. doi.org/10.3233/HIS-2007-4304
• [304] Weight Agnostic Neural Networks — Gaier & Ha, NeurIPS, 2019. arxiv.org/abs/1906.04358
• [305] Abandoning Objectives: Evolution through the Search for Novelty Alone — Lehman & Stanley, Evolutionary Computation, 2011. doi.org/10.1162/EVCO_a_00025
• [306] Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning — Such et al., 2017. arxiv.org/abs/1712.06567
• [307] Population Based Training of Neural Networks — Jaderberg et al., 2017. arxiv.org/abs/1711.09846
• [308] Model-Agnostic Meta-Learning (MAML) — Finn, Abbeel & Levine, ICML, 2017. arxiv.org/abs/1703.03400
• [309] On First-Order Meta-Learning Algorithms (Reptile) — Nichol, Achiam & Schulman, 2018. arxiv.org/abs/1803.02999
• [310] Meta-SGD: Learning to Learn Quickly for Few-Shot Learning — Li et al., 2017. arxiv.org/abs/1707.09895
• [311] Meta-Learning with Differentiable Convex Optimization (MetaOptNet) — Lee et al., ICLR, 2019. arxiv.org/abs/1804.03158
• [312] Meta-Learning with Latent Embedding Optimization (LEO) — Rusu et al., ICLR, 2019. arxiv.org/abs/1807.05960
• [313] A Simple Neural Attentive Meta-Learner (SNAIL) — Mishra et al., ICLR, 2018. arxiv.org/abs/1707.03141
• [314] One-Shot Learning with Memory-Augmented Neural Networks (MANN) — Santoro et al., 2016. arxiv.org/abs/1605.06065
• [315] HyperNetworks — Ha, Dai & Le, ICLR, 2017. arxiv.org/abs/1609.09106
• [316] LoRA: Low-Rank Adaptation of Large Language Models — Hu et al., ICLR, 2022. arxiv.org/abs/2106.09685
• [317] QLoRA: Efficient Finetuning of Quantized LLMs — Dettmers et al., NeurIPS, 2023. arxiv.org/abs/2305.14314
• [318] Dynamic Convolution: Attention over Convolution Kernels — Chen et al., 2019. arxiv.org/abs/1912.03458
• [319] Parameter-Efficient Transfer Learning for NLP — Houlsby et al., 2019. arxiv.org/abs/1902.00751
• [320] AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning — Zhang et al., ICLR, 2023. arxiv.org/abs/2303.10512
• [321] Prefix-Tuning: Optimizing Continuous Prompts for Generation — Li & Liang, ACL, 2021. arxiv.org/abs/2101.00190
• [322] The Power of Scale for Parameter-Efficient Prompt Tuning — Lester, Al-Rfou & Constant, ACL, 2021. arxiv.org/abs/2104.08691
• [323] P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning — Liu et al., 2021. arxiv.org/abs/2110.07602
• [324] Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (IA³) — Liu et al., ICLR, 2023. arxiv.org/abs/2205.05638
• [325] Physics-Informed Neural Networks — Raissi, Perdikaris & Karniadakis, Journal of Computational Physics, 2019. doi.org
• [326] Learning Nonlinear Operators via DeepONet — Lu et al., Nature Machine Intelligence, 2021. arxiv.org/abs/1910.03193
• [327] Fourier Neural Operator for Parametric PDEs (FNO) — Li et al., ICLR, 2021. arxiv.org/abs/2010.08895
• [328] fPINNs: Fractional Physics-Informed Neural Networks — Pang, Lu & Karniadakis, SIAM J. Sci. Comput., 2019. arxiv.org/abs/1811.08967
• [329] hp-VPINNs: Variational Physics-Informed Neural Networks With Domain Decomposition — Kharazmi, Zhang & Karniadakis, 2020. arxiv.org/abs/2003.05385
• [330] Message Passing Neural PDE Solvers — Stachenfeld et al. (DeepMind), ICLR, 2022. arxiv.org/abs/2002.05674
• [331] Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions (Tacotron 2) — Shen et al., ICASSP, 2018. arxiv.org/abs/1712.05884
• [332] Conformer: Convolution-augmented Transformer for Speech Recognition — Gulati et al., Interspeech, 2020. arxiv.org/abs/2005.08100
• [333] Robust Speech Recognition via Large-Scale Weak Supervision (Whisper) — Radford et al., 2022. arxiv.org/abs/2212.04356
• [334] Deep Speech 2: End-to-End Speech Recognition in English and Mandarin — Amodei et al., ICML, 2016. arxiv.org/abs/1512.02595
• [335] FastSpeech: Fast, Robust and Controllable Text to Speech — Ren et al., NeurIPS, 2019. arxiv.org/abs/1905.09263
• [336] Music Transformer — Huang et al. (Google), ICML, 2019. arxiv.org/abs/1809.04281
• [337] Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation — Luo & Mesgarani, IEEE/ACM TASLP, 2019. arxiv.org/abs/1809.07454
• [338] All-Optical Machine Learning Using Diffractive Deep Neural Networks (D2NN) — Lin et al., Science, 2018. doi.org
• [339] Deep Learning with Coherent Nanophotonic Circuits — Shen et al., Nature Photonics, 2017. nature.com
• [340] Highly Accurate Protein Structure Prediction with AlphaFold — Jumper et al., Nature, 2021. nature.com
• [341] TabNet: Attentive Interpretable Tabular Learning — Arik & Pfister, AAAI, 2021. arxiv.org/abs/1908.07442
• [342] RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control — Brohan et al., 2023. arxiv.org/abs/2307.15818
• [343] CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code — Wang et al., EMNLP, 2021. arxiv.org/abs/2109.00859
• [344] An End-to-End Trainable Neural Network for Image-based Sequence Recognition (CRNN) — Shi, Bai & Yao, IEEE TPAMI, 2017. arxiv.org/abs/1507.05717
• [345] Neural Machine Translation by Jointly Learning to Align and Translate — Bahdanau, Cho & Bengio, ICLR, 2015. arxiv.org/abs/1409.0473