Research & TrendsDeep Dive (Vale)
A nautilus in a naval coat holds a lantern whose narrow beam lights only a few branches of a tall tree-shaped cabinet of numbered drawers, while drawers on the dark lower branches hold small boutique labels.

A Generative Recommender Cannot Rank a Product It Never Spells

Semantic-ID recommenders must write out a product's code before they can score it, and a 9 October arXiv paper measures how often a finite beam stops that happening. Its fix scores the whole catalogue and lifts accuracy, but rarely rated products are still missed and top-ten reach fell on one dataset, so retailers should ask vendors for catalogue reach by sales quartile.

Neritus Vale

A generative recommender built on semantic IDs has to spell a product’s code before it can rank it, and a product whose code the beam never writes is not scored for that shopper. In “Beyond Semantic IDs”, submitted to arXiv on 9 October, Yu Sun and eight co-authors call the resulting limit a “finite-beam reachability ceiling” and measure it. Their results show a ceiling set by choices a retailer never sees, and a fix that raises accuracy without reliably widening what shoppers are shown.

Semantic IDs replace a product’s arbitrary catalogue number with a short code in which similar products share the opening numbers. DoorDash’s Sudeep Das likens them to “postal codes for your items”. Recommenders of the TIGER type predict a shopper’s next product by writing its code one number at a time. A beam search keeps only the most promising partial codes at each step and drops the rest. A product whose opening numbers are dropped is never written out, so the model never scores it.

For a given trained model and beam budget, the ceiling caps how often a user’s next pick can appear in the returned list, even when its code is valid. The authors note that “a target that never enters the candidate set cannot be recovered later”, and they say it of a reranker that sees only the decoded beam. If merchandising rules run after retrieval, a boost for a new label acts only on what the beam returned.

On MovieLens-1M, a public film-rating dataset, the authors’ TIGER-style model left 33.3% of the films users rated next out of its 100-item lists. Doubling the list, which takes a wider beam, cut the misses to under a quarter; the authors read this as “a finite-compute trade-off, not an architecture-invariant catalog ceiling”. The continuous model described below misses 32.7% at 100 items, and the paper’s text does not separate beam exclusion from ordinary ranking error. Beam width is a cost line for the vendor and a reach limit for the retailer.

The misses fall hardest on products with the least history. The authors compute each film’s hit rate, then average it within quartiles of training frequency. At 100 items and seed 42, the code model’s hit rate was 25.8% in the quarter of films with the fewest training ratings and 68.6% in the quarter with the most. A separate team reported that most missed targets are pruned “within the first two decoding steps”; its analysis is preliminary and covers Beauty, Instruments and Yelp. The quartiles track how often a film was rated, not how new it is, so the fashion reading is an analogy: a young label’s first collection would start in the rarely bought quarter of a marketplace.

Semantic IDs are pitched as a friend to the long tail. TIGER’s abstract reports improved retrieval for items with no prior interaction history, and the recommendation vendor Shaped lists the codes as coming soon, “designed to improve generalization on new and long-tail items without sacrificing overall ranking quality”. Its wording closely tracks a 2023 study of YouTube’s ranking model.

The other evidence is less direct. In a 21-day online test, DoorDash found that a ranking update combining semantic-ID features with other changes cut the average historical popularity of its top carousel product by 18.1%. Both the YouTube and DoorDash studies fed the codes to a ranker that still scores every candidate. TIGER claimed cold-item gains for the model that writes the codes, and “Beyond Semantic IDs” leaves cold-item accuracy unmeasured, so the claim is untested here rather than refuted.

The paper’s fix removes the spelling. Its CQG-Single model turns a user’s history into one continuous query and scores the whole index against it. It beat the TIGER-style baseline in all seven training runs on the film data, with 55.3% as many trainable parameters. That is dense retrieval, the approach generative decoders were sold as replacing, and the authors do not claim it as new.

A product can be scoreable and still be shown to no one.

The strongest objection blames short lists, not the beam, for the missing tail. For that to hold, removing the beam would have to leave slow sellers where they were, and on the film data it nearly did. The continuous model found next picks from the bottom quarter 26.4% of the time, barely more often than the code model, and showed the same gradient up to 70.0% in the top quarter. One result went the other way: at shallow depths the code model stayed stronger for some rare films, though the authors report that Beauty does not reproduce it. The objection explains the accuracy numbers; it cannot say which cutoff, the beam or the ranking, hides a given product.

Reach was not the code model’s weak spot. Pooled across users at seed 42, its top-100 lists touched 97.0% of films, 96.3% of beauty products and 99.5% of sports products. The ceiling therefore bites shopper by shopper, not catalogue-wide.

Removing the beam did not reliably widen what users saw, either. On the sports data the continuous model was more accurate, but its top-ten lists, pooled across users at seed 42, touched 63.1% of the catalogue against the code model’s 94.0%. The paper’s text mentions only the coverage gains on beauty and, more modestly, on films.

Reach also depends on how the codes are built. A second build, closer to the original TIGER recipe, put 44.0% ± 13.7% of the beauty catalogue into at least one user’s top 100 (a three-seed mean), and 14.0% ± 7.6% of the least-reviewed quarter. The authors say this build ranks below the reference TIGER and that its figures must not be generalised to a fully reproduced TIGER checkpoint. It still shows that reach belongs to a particular build, not to the method.

Shopee’s published decoder points the same way. UniRec, deployed on Shopee’s marketplace, states its aims as suppressing token collapse and balancing traffic across codebook entries, and it adds “exposure-weighted capacity penalties” to curb a “hereditary Matthew effect”, the pull toward products that already sell. If that is how a production system is tuned, a retailer that cannot see where such settings sit cannot assume the tail survives them. That is an inference from one system, not a finding of the paper.

The paper is weakest on the two things that define apparel catalogues: size and constant newness. Its three datasets are small and public, the largest holding 12,101 beauty products, and one of them is films. Scoring is offline, with no production traffic: each user’s last interaction is hidden and the model must find it. The headline tables use a single seed, 42. On beauty the two models’ 95% intervals overlap, and the film gain of 0.0107 Recall@10 carries an interval of 0.0021 to 0.0193.

The authors call their baseline weaker than published systems and say production latency, memory, index-update cost and accuracy on new items “remain unmeasured”. Their fix trains against the whole catalogue at every step, a cost that grows with the catalogue and was tested only at these sizes. A sampled-negative variant is cheaper, and the authors say it retains performance close to the TIGER-style baseline. A full-rank SASRec baseline also scored higher than the continuous model on the sports data.

The paper’s useful move is to split one question into two. The first is how often the model finds what a shopper wanted; the second is how much of the catalogue it shows to anyone. Vendors should report the second on the retailer’s own catalogue, by sales quartile, at the site’s list length and the live beam width. If a vendor can only report the first, the retailer is buying accuracy and assuming reach. Reach can be written into the contract instead.