
Shopify Researchers Traced the Generative Recommender's Edge to Popularity
A Shopify paper finds that popularity is the largest part of a semantic-ID recommender's lead over dense retrieval on 'value', and that one static score per product hands that lead to the dense retriever. On internal data a 14-billion-parameter model also lost to a far smaller encoder, so retailers should make generative systems beat that cheap baseline on new-season stock before paying for them.
Neritus Vale
Popularity is the largest part of the lead a generative recommender held over its dense rival on “best relevant item value”, Shopify researchers report. Their paper, posted to arXiv on 13 October by Renbo Tu, Yang Liu and Shuying Sun, shows that one static score per product, added to the dense retriever, reverses that lead. On internal e-commerce data, a semantic-ID model with 14 billion parameters also lost to an encoder about 47 times smaller.
The two methods find candidates differently. Generative retrieval writes out a product’s semantic ID one token at a time. In dense retrieval, a long-standing standard, shopper history and catalogue share one embedding space, and the retriever returns the nearest products. On public data, the authors read both from one jointly trained model built on LIGER, a hybrid of the two, so the arms “differ mainly in their retrieval readout”.
Which method wins depends on what is scored. On Amazon’s Toys & Games reviews, the dense readout found each user’s next product more often at 50 results, by more than a third on the 2014 release. The generative readout won on one score, which compares the most valuable same-category product in each list.
Value, in this paper, has no price in it. Each product gets one static score from catalogue percentiles: half “brand footprint”, the rest split equally between popularity and a star rating discounted for thin review counts. The authors call it “a transparent proxy for item value”; no margin, stock or sell-through figure enters it. On this measure, the two lists’ best products came within 0.05 of each other for about seven in ten of the users both methods covered.
Popularity did more of the work than its weight implies. Among users whose generative list won, it supplied 49.8% of the lead on the 2014 data and 65.4% on the far larger 2023 catalogue.
The authors offer repeated exposure as a possible explanation: a model trained to predict product codes sees some far more often than others, and may learn to prefer them. They cite a Spearman correlation of 0.582 in support.
The authors then gave the dense retriever the same score. Their DR+prior method, which needs “no retraining or decoder at serving time”, reranks the retriever’s own top 200 candidates by an even average of similarity and product score. On the 2014 data, the best-item deficit became a lead of 0.023 on a zero-to-one scale; on 2023 the lead was 0.006. Its lists still beat the generative lists on recall, and scored highest on whole-list value on both datasets.
Nothing in that score knows who is shopping.
The reversal is partly built in. The prior adds the very score the value metric counts, and the authors frame the test accordingly: it “checks whether an explicit catalog prior can reproduce GR’s measured lead”. It shows that the lead needs no decoder.
The abstract’s size comparison comes from a separate, looser test. On internal e-commerce data, a Qwen3 model that writes semantic IDs faced a GTE encoder of about 300 million parameters, and scored 0.453 to the encoder’s 0.474 on NDCG@50 (n=5,998), with no interval reported. Because the two differ in backbone as well as size, the paper says the test “cannot isolate the effect of retrieval architecture”.
The paper spends one sentence on the language-model judge behind the industrial scores. The judge was “calibrated with 250 human labels” and grades each candidate from zero to one; no judge model or agreement rate is reported. Under it, the static score lifted the encoder a further 11.8%. The paper cites earlier work that traces generative popularity bias to “inherent bias within LLMs”; its own explanation is repeated exposure in training. It never tests whether its judge prefers popular products.
Earlier coverage here pointed the other way, though on different evidence. In May we wrote that papers from Meta and Shopee “retire two-tower as the default for any catalogue large enough that personalisation moves margin”. Those papers concerned serving infrastructure, not value metrics. In February, a Shopify engineering post bylined Yang Liu and Ali Khanafer called semantic IDs a “promising” direction; the paper does not cite it. Tu, Liu and Sun locate part of the generative edge in the catalogue, where an incumbent retriever can reach it. If that holds beyond toys, the incumbent has more life in it than our May headline allowed.
The strongest objection is that popularity is no flaw in retail, since a model that learns which best-sellers suit a given shopper is personalising. It holds only if those picks depend on each shopper’s history in ways one store-wide score cannot copy. The paper cannot confirm that: its metric counts the same score the prior supplies, so it registers no personal fit. What it does show is that the measured lead survives without any shopper-specific signal.
The fix carries a price, and a fashion catalogue would pay it. The prior lowered the dense retriever’s Recall@50 on 2014 from 0.228 to 0.210. Of 296 test products unseen in training, the generative model found none, a related reachability limit we covered. The prior cut the dense retriever’s finds among such products from 47 to 36 on 2014 and from 27 to 21 on 2023. A store that drops a collection every season lives on products with no history. A score built half on popularity and ratings will probably rank them low, though the brand-footprint half may lift new-season stock from established labels.
For a fashion buyer, the evidence is narrow. The public data are toy reviews, and a review is not a purchase. The paper does not say whose shoppers or catalogue fill the industrial dataset, and the released code reproduces only the public results. It also pits the methods head to head, though Shopify’s February post says a production recommender “is usually an ensemble”, where a retriever earns its place by finding what others miss.
For retailers, the paper turns a sales pitch into a test. Ask what the vendor’s “value” metric contains, and whether the model still wins on new-season stock once the current retriever gets the same product score. The authors call dense retrieval with a static prior “a useful baseline for future GR systems”. A model that cannot clear it has bought parameters to rediscover the best-seller list; the cheaper move is a product score and one reranking step.