
Shuffle a Small Shop's Catalogue and the AI Picks Something Else
Google's RPTune paper finds that a long-context model can read a small shop's entire catalogue, but its pick then depends on the order it reads in, and a learned curator that keeps a quarter of the list and seats the best candidates last beat the full catalogue for every model at every shop tested. If assistants built this way reach small apparel shops, the running order of the prompt does the work of shelf placement, and the merchant does not set it.
Admiral Neritus Vale
Give a language model a small cosmetics shop’s whole catalogue, ask for a long-wearing berry lipstick, then shuffle the list and ask again, and it may pick a different product. RPTune, posted to arXiv this week by Google researchers including Chuxuan Hu and Sercan Ö. Arık, finds that small shops can skip the retrieval stage and that the model’s pick then depends on the order it reads in. If assistants built this way reach small shops, the prompt’s running order will work as shelf placement, set by whoever trains the curator that arranges it.
On the paper’s own benchmark, reading the whole shop beat retrieval. Across the Shopify storefronts the team analysed, the median catalogue came to 51,000 tokens, short enough to show a model every product at once. Retrieval shows it a shortlist instead, which can omit the right product, the failure we examined on 25 September. Given the full catalogue, Gemini 3.7 Flash named exactly the right variant at least nine percentage points more often than any search pipeline the team tried, on average.
Models read a long catalogue unevenly, and the benchmark was built on that weakness. The 2023 study Lost in the Middle found they often use the middle of a long input less well than its ends. Gemini 3.1 Pro wrote the test’s shopper queries, then answered each over repeatedly shuffled catalogues. The authors kept the 31.9% of candidate queries on which its favourite answer won a moderate share of the runs, and made that favourite the answer key, with human experts settling ties. The headline results therefore rest on synthetic queries chosen because order changes the pick. On a set of order-stable queries from Beauty Bakerie, the full RPTune pipeline, curation plus post-training, still removed roughly 80% of a small Gemma model’s errors.
RPTune’s answer is a curator that decides what the model reads and in what order. An encoder scores every product against the query, and a small layer trained against the picks of Gemma, a compact Google model, adjusts the scores. The top quarter then enters the prompt in ascending order, strongest candidates last, nearest the instruction to choose. Across nine models, the curated quarter added 13.5 points of exact-match accuracy on average and helped every model at every shop. The system that wins is a retrieval stage with a longer shortlist and a seating plan.
The abstract’s “up to 31.4 percentage points” comes from one pairing, Grok 4.1 Fast run without reasoning on Beauty Bakerie’s cosmetics catalogue. It is the best curation-only cell in the results table. The results section’s maximum of 20 points is a different measure, the best model’s average across all seven shops. Neither describes a typical shop.
On one shop, the paper isolates how much of the curator’s work is placement. For Beauty Bakerie, a chart in the paper credits 19.7 points to “reordering alone”: the same quarter, shuffled and then sorted, read by RPTune’s post-trained Gemma. The paper does not separate this from habit, and that model was trained on sorted lists, so part of the gap may be learned preference. The untuned model also did best with the strongest candidates placed last. In this prompt, eye level is the bottom of the list.
Generic relevance ranking is a weaker stand-in for a curator trained on the reader’s picks. Pruning to the same quarter with Jina Reranker 3.5 left Gemini 3.7 Flash barely ahead of the full catalogue on average, 37.8% against 36.8%, and worse at five of the seven shops. In an ablation that swapped in a supervised objective, the authors say the gains “come from optimizing against the downstream LLM’s selections”.
Cutting to a quarter revives a milder shortlist problem: the right variant survives in 77.1% of curated lists on average. Fine-tuning then rewards the model for picking the best product left in its list, whether or not the shop’s best survived. It learns to sell what is on the rail.
RPTune’s planogram is drawn from Gemini’s relevance scores and Gemma’s reading habits, and the merchant contributes nothing but the catalogue.
Even curated, exact variants stay hard to find. Curation plus fine-tuning raised the small Gemma model’s exact-match rate from 10.7% to 31.0%. The misses were close substitutes: on one Beauty Bakerie query, the model sometimes chose a different matte lip whip with the same price and weight as the reference, told apart only by shade name and description.
Exact match is also a strict test, and the answer key is not beyond dispute. Four non-Gemini models acting as judges accepted it for only 64.9% of sampled queries, and the authors caution that Gemini 3.1 Pro’s labels “may favor Gemini-family backbones”.
The seven shops sell cosmetics, breakfast food, pet food, skin and hair care, eyewear, zero-waste goods and “custom gears”. Clothing is absent, though Store Leads ranks apparel as Shopify’s largest category. The study’s most variant-heavy merchant sells eyewear, with 739 variants of 186 products. A shirt in five sizes and three colours is fifteen near-identical variants on its own. If small apparel shops hand their catalogues to such assistants, the retrieval stage may go, but the lip-whip problem will arrive in every colourway.
The strongest objection is that order matters only to weak readers, so better models will make the curator redundant. For that to hold, the gain from curation would have to shrink towards zero as models improve. The table does not show that: Claude Opus 5 gained least of the frozen models, only 0.6 points at Beauty Bakerie, yet it still gained at every shop. The curator also cut response times for all nine models, by 28% on average. The paper measures latency, not cost, but a shop has reason to send the shorter prompt even to a perfect reader. Whoever picks and orders that quarter owns the shelf.
Anyone can read a Shopify merchant’s catalogue: the authors pulled all seven from a public endpoint that, they note, needs no “scraping, credentials or merchant cooperation”. Arranging it falls to whoever builds the assistant. If curators are trained like RPTune’s, on synthetic shoppers and model judgements, the best spot in each prompt would go to whatever those models judge a match. RPTune uses no behaviour logs, so the one input it lacks is what the shop’s customers bought. The merchant can supply it, or leave the shelf to whoever builds the assistant.