Research & TrendsDeep Dive (Vale)
A long paper list unrolls down a library ladder; its top entries are gilded, lower rows have gaps where cards were snipped out, a black ink cartridge tumbles from one gap, and a nautilus with a ruler measures far down the list while two near-identical pairs of jeans hang untouched on a hook.

Tuned for the Top Ten, Recommenders Lost Ground in the Hundreds They Hand to Rankers

In an HKUST and Trip.com preprint, reinforcement learning sharpened generative recommenders' first results on four of five Amazon review datasets but cut recall at 500 on all five, within the hundreds of candidates such models hand to rankers. The authors trace the loss to a reward that treats every sampled item but the logged one as wrong; a one-term fix largely restored recall offline. No deployed system or clothing category was tested.

Neritus Vale

Generative recommenders now routinely get a final round of reinforcement learning, judged mostly at the top of the list. A preprint from HKUST and Trip.com Group, posted on 17 November, looks further down, at the hundreds of candidates such models hand to rankers. The authors say RL left the model “a better ranker and a worse retriever” on all five Amazon review datasets. Their tables show the top-ten gain on four of the five and the loss at 500 on all of them.

After supervised fine-tuning (SFT), the RL step draws 16 candidates per user history; when the logged next item is among them, it rewards that item and pushes the rest down. Every result comes from one model, Qwen2.5-1.5B, trained with one GRPO recipe. In the paper’s results, hit@K is the share of test users whose next item ranks in the top K, in per cent; changes are in points. On Sports, hit@10 fell 0.18 points; only hit@5 rose (+0.20).

Dataset hit@10, SFT → RL hit@500, SFT → RL hit@500 with fix
Office 15.97 → 16.41 (+0.44) 52.84 → 47.69 (−5.14) 52.14
Industrial 13.19 → 13.70 (+0.51) 53.23 → 51.72 (−1.51) 53.63
Beauty 4.80 → 5.27 (+0.47) 37.16 → 36.67 (−0.49) 37.39
Toys 5.00 → 5.50 (+0.50) 35.18 → 35.14 (−0.04) 35.60
Sports 3.70 → 3.52 (−0.18) 29.88 → 29.21 (−0.67) 30.11

The gain sits at the top, the loss below. On Office the gain turned into a loss by about the 20th result, and on Industrial by about the 50th. Both are shallower than any production list the paper surveys.

A near-flat total can hide a reshuffle. Toys barely moved at the deepest cut, yet on every dataset at least 14% of the targets the SFT model ranked there fell out after RL while other users’ targets moved in. That changes whose next item would reach a ranker.

The authors trace the loss to the label. With a single logged next item per user, the reward treats every other sampled candidate as wrong, though these are often “the next items of other users with similar histories”. In the paper’s example, black ink sampled beside a user’s colour ink is penalised, then sinks, via the shared network, for a similar user who chose it next.

A reward that knows only the logged answer counts every other plausible answer sampled alongside it as a mistake.

The ablations point the same way. On Office and Industrial, in single runs with a synced reference model, removing the penalties removed both the loss and the gain. In a single Office run, users whose next item would draw the heaviest penalties lost 23.4 points at the deepest cut.

Published tests of the RL step mostly sit where the gain is. Kuaishou’s OneSearch-V2, the paper notes, compares its models with and without RL on the first ten results. On the paper’s own models, the standard check “sees the gain and none of the loss”.

Industrial lists run far deeper. The paper’s survey gives these systems’ post-training and candidates per request; Kuaishou’s are end-to-end generators, also run as retrieval for a reward model. None was tested in the study.

System Post-training Candidates
OneRec (Kuaishou) DPO, ECPO 512
OneRec-V2 (Kuaishou) GBPO 512
OneSearch (Kuaishou) DPO 512
OneSearch-V2 (Kuaishou) GRPO 512
TSGR (Taobao) GRPO (tested, not deployed) 1,500
GrowthGR (Taobao) GRPO 1,000
EG-GRPO (Tmall) GRPO 100
GenR-PO (JD) DPO 100
PinGER (Pinterest) DPO 100
GenRec (JD) GRPO not stated

Shopify’s report, covered here on 9 November, shows a related problem with a different label. Its search models were tuned on a relevance judge’s scores, by supervised learning, DPO or GRPO, and that raised judged quality while lowering purchase recall in the top ten. The authors blame exposure bias in the metric, which can retrieve “alternatives that receive no credit under purchase recall”. They add that the decline “does not establish whether these alternatives would lead to more or fewer purchases if shown”. In the HKUST study the reward punishes plausible alternatives; at Shopify the metric cannot credit them.

The fix, out-of-rollout anchoring, adds a single loss term. GRPO’s own KL tether misses the falling items, because it is computed only on the rollout. For each prompt the anchor draws 16 items from outside the rollout, near neighbours of the candidates that share their leading semantic-ID codes, and holds the policy to the SFT model’s ordering of them. The table’s last column shows how much recall returns. At the top, the anchored model did not differ significantly from GRPO in any seed on any dataset, and the paper reports no training time.

None of the sets is a clothing category. Beauty, Toys and Sports come from Amazon review data the paper cites to McAuley et al. (2015); that paper is about which clothes go together, and its abstract’s example of alternatives is “two pairs of jeans”. If the mechanism carries over to apparel, the substitutes the reward punishes would often be what similar shoppers buy.

The strongest objection is scale. These catalogues are small: the deepest cut covers 14% of the smallest, and the larger catalogues lost under a point.

The paper itself qualifies that comparison. Its hyperparameters were not tuned for Beauty, Toys and Sports, their SFT models are weaker, and their rollouts contain the target only 11–24% of the time, so the smaller losses may reflect untuned RL, not catalogue size. Production rewards also differ: GR4AD-X, the Kuaishou ad generator we covered on 2 November, is rewarded with revenue estimates, not a logged next item. For the thesis to fail, the pressure behind the loss would have to fade at production scale or under such rewards.

Two industrial papers report symptoms consistent with that pressure, though HKUST treats them as single observations. Pinterest’s PinGER, tuned by DPO on its ranker’s preferences, says recall “collapses within a few thousand steps” when the DPO term runs alone. Taobao’s TSGR, rewarded by a ranker’s top picks, masks the rewards of its middle candidates to keep GRPO from “mistakenly suppressing ambiguous candidates” that may belong in its top 1,000. Plain GRPO there added 0.02% to HR@1000; the version with reward masking and DPO augmentation added about 0.5%, which the authors judged too marginal to deploy.

If the effect holds in production, an RL step tethered only at the top trades shoppers in the middle of the list for a better top-ten score. The authors recommend that RL be “evaluated at the depth at which the model is deployed”. Which depth to report is the team’s choice; the shoppers in the middle get no say.