Research & TrendsDeep Dive (Vale)
A brass automaton shop assistant reaches for the first of five near-identical black coats on a numbered rail, while a nautilus in a naval coat shuffles a fan of number tags at the far end.

The Two Steadiest of Eight AI Shop Assistants Leaned Hardest on the First Slot

In a University of Granada test, eight open-weight models acting as shop assistants all changed their picks when the same shortlist was shuffled, and OpenAI's GPT-OSS pair, the two steadiest and the least likely to stray to a worse product, gave the first slot the most picks. On a short list the slot mostly settled ties between products a model rated alike: a ranking factor that a relevance check cannot catch and that OpenAI's account of how ChatGPT picks products does not mention.

Neritus Vale

In a University of Granada test of eight AI shop assistants, the two that changed their pick least when a shortlist was shuffled gave the first slot the most picks. OpenAI’s open-weight GPT-OSS models seldom let the order push them to a clearly worse product. They used it to choose between products they rated alike: position bias that passes a relevance check and still picks the winner.

The study, posted on 27 October by Carlos Peláez-González, Francisco Herrera and Andrés Herrera-Poyatos, rebuilt a common design for AI recommenders. An embedding model fetched the five products whose titles best matched a request, and a language model told “You are a shop assistant” recommended one. Each request went to each model with its shortlist in 20 random orders.

The eight models were Meta’s Llama 3.1 8B and 70B and Llama 4 Scout, OpenAI’s GPT-OSS 20B and 120B, and DeepSeek R1 distilled into 7B, 8B and 32B models. All are open-weight; no closed commercial model was tested. The 2,000 requests were written from templates, as in “Show me reputable brand fitness trackers.”

The products were real. The authors sampled 50,000 from the catalogue of the Amazon ML Challenge 2023, a hackathon on predicting product length, and say it spans electronics, clothing, food and personal care. Fashion was one of 18 request categories, and the team’s query generator lists summer dresses, jeans and winter jackets among its products. The paper breaks out no result by category, so none of its findings is specific to apparel.

Every model failed both of the paper’s statistical tests for positional independence, and order alone moved picks often. GPT-OSS 120B departed from its most frequent choice for a request in 15.5% of shuffles, the lowest rate in the study. Nothing else about the shortlist had changed. DeepSeek R1 7B departed in 34.0%.

Which slot benefited depended on the model. With the order random, a position-blind model would give each slot a fifth of its picks; GPT-OSS 20B gave the first slot 25.4%. Llama 3.1 70B, at 23.0%, also leaned to the first slot, so the habit is not OpenAI’s alone, though the GPT-OSS pair led. Llama 4 Scout avoided the first slot, and on longer lists its preference for later ones “grows markedly”.

The authors separate soft bias, where the slot decides between comparably relevant products, from strong bias, where it pushes the pick towards a clearly less relevant one. They build that relevance set from each model’s own pick frequencies, so “less relevant” means outside what that model usually picks. Soft bias, they write, “does not degrade the relevance of the recommendation, but it still makes the exposure of comparably relevant items depend on the prompt construction rather than on their content.”

On the standard shortlist, strong bias was small and close to the noise floor. Its highest rate in the main run was 9.8% of picks, for DeepSeek R1 7B, and the authors read every such figure as an upper bound. Re-running one fixed order with sampling randomness alone produced comparable rates.

Adding shuffles raised strong bias by 0.3 to 1.1 points for six models. For the GPT-OSS pair the fixed-order comparison found no significant difference, though the authors say it prevents a fully clean comparison, because the fixed order puts the retriever’s top item in slot 1, which those models favour. A second control, with random item labels, found shuffling raised strong bias by 0.5 to 1.3 points for all eight.

The authors call the GPT-OSS pair’s preference for the first slots a tie-breaker that “acts as a tie-breaker among comparably relevant candidates”. If the models behind commercial assistants behave like these two, their position bias will hide in the choice between good answers.

No relevance check can catch a tie-break, since every tie-break returns a relevant product.

Coveo’s release on buying Anglera, which we examined earlier today, argues that assistants reward richer product data and offers a test as evidence. The Granada test held every title and description fixed and still moved the pick. Content decides which products are in contention; among those, the slot can choose the winner.

Longer shortlists made every model less stable and, on the authors’ measure, pushed more picks outside the set of comparably relevant products. With 20 candidates, Llama 4 Scout’s strong-bias rate reached 36.6%, the paper’s highest. The noise floor was measured only on the short list, so how much of that is position is unknown. Raising the sampling temperature, tested on the short list, changed little.

An unshuffled prompt hands the first slot to retrieval rank. This pipeline ranks by how closely each title matches the request, so the closest title is both first in the list and first in the prompt. A model with GPT-OSS’s first-slot habit would give that title the ties, and it would win twice: once to make the list and again to top it. The seller chooses neither the model nor its habit.

The strongest objection is that none of this need reach a real assistant. The eight models are open-weight, and the assistants shoppers use may run on something else. The authors’ remedy, shuffling the candidates before each answer, “prevents items from being systematically favored by their retrieval rank at no additional cost”. For the thesis to fail, those assistants would need models with no slot preference, or lists shuffled every time.

Neither condition can be confirmed from outside, and the first already looks doubtful. In an audit we covered on 20 October, all seven models in the control, GPT-4o and Claude Sonnet 4.6 among them, favoured the first of two identical product cards when the prices matched. Identical cards are a perfect tie, and the slot settled it.

Whether a list is shuffled is a design choice, and OpenAI does not say how ChatGPT makes it. Its help page says product results “are not ads” and lists what the model considers, among other things price and descriptions, the query and context, reviews, and model responses generated before it considers new search results. It says nothing about the order candidates arrive in.

The audit is cheap, and only an assistant’s builder can run it. In the authors’ simulation, fewer than 200 requests, each shuffled 20 times, usually sufficed to catch a bias as weak as the weakest they measured, though real ones may need more. A brand cannot reorder ChatGPT’s prompt; a retailer running its own assistant can. Whoever fills the prompt decides whether habit or a shuffle settles the ties, and that is a ranking decision whether or not anyone writes it down.