Neural Networks Taxonomy - 16. Recommendation and Retrieval Networks

Details
Written by: Babak
Category: Blog
Hits: 526

Article Index

Page 17 of 29

16. Recommendation & Retrieval Networks

A recommender predicts which items a user is likely to engage with. The dominant modern design is the two-tower retrieval network in Fig 18: a user tower and an item tower each map their features into the same embedding space, and relevance is simply the dot product of the two embeddings — so millions of candidate items can be scored by fast nearest-neighbour search.

User features into a user tower producing a user embedding, item features into an item tower producing an item embedding, the two embeddings meeting at a dot-product score
Fig 18. A two-tower recommender encodes users and items separately, then ranks by the dot product of their embeddings.

Origins and rise

Wide & Deep paired a memorising linear model with a generalising deep network.[230] Neural Collaborative Filtering replaced the classic matrix-factorisation dot product with a learned network over user and item embeddings,[231] and DLRM scaled the recipe to industrial click-through prediction with enormous embedding tables.[232]

Modelling feature interactions

A click-through model lives or dies on how well it combines sparse categorical features, so a family of models extends the Wide & Deep template[230] with explicit interaction machinery. DeepFM fused a factorisation machine with a deep network over a shared embedding table, learning low- and high-order feature crosses jointly.[233] Deep & Cross Network replaced the memorising wide side with a cross network that builds bounded-degree feature crosses at every layer,[234] and its successor DCN V2 made those crossing matrices low-rank so they scale to web-scale ranking.[235] xDeepFM added a compressed interaction network that crosses features at the vector rather than the bit level,[236] while AutoInt used multi-head self-attention to learn automatically which feature combinations matter.[237]

Sequential recommenders

Rather than treat a user as a static bag of preferences, sequential models predict the next item from the ordered history of interactions. GRU4Rec first applied a recurrent network to anonymous session data,[238] then SASRec swapped the recurrence for a causal self-attention stack that captures long-range dependencies more cheaply.[239] BERT4Rec made the attention bidirectional, training with a masked-item objective borrowed from BERT,[240] and SR-GNN modelled each session as a graph, propagating with a graph neural network to capture more complex item transitions.[241]

Two-tower retrieval

At web scale a model cannot score every item for every request, so candidate retrieval is split into two independent encoders like those in Fig 18. DSSM introduced this design for web search, mapping queries and documents into a shared semantic space where relevance is a cosine similarity.[242] The YouTube recommender scaled the idea to hundreds of millions of videos with a deep candidate-generation tower whose output embeddings are served by approximate nearest-neighbour search,[243] the factorisation that lets the item tower be precomputed and indexed offline.

Applications

  • Product, video, and feed recommendation at web scale.
  • Advertising click-through and conversion prediction.
  • Candidate retrieval via approximate nearest-neighbour search.

Strengths and limitations

Strengths Limitations
Score millions of items via precomputed embeddings. Cold start: little signal for new users or items.
Learn feature interactions automatically. Sparse embedding tables dominate memory.
Decouple offline indexing from online serving. Feedback loops can amplify popularity bias.