Agentic CommerceDeep Dive (Vale)
Ten small robot shoppers with effort dials on their chests turned to high all reach for the same wrong-size jumper on a clothes rail, each holding up a scorecard reading 10, while the right-size jumper hangs untouched and a nautilus in reading glasses watches with a clipboard.

Set to Think Harder, Most Shopping Agents Grew Surer and Still Made the Same Mistakes

In a University of Chicago preprint testing ten frontier models on fixed lists of ten products, raising reasoning effort from low to high significantly raised stated confidence in 17 of 30 model-benchmark comparisons and accuracy in one, after correction for multiple testing. The models also failed together: on WebShop at low effort, 47 of 493 tasks beat all ten, where independent errors would predict practically none.

Neritus Vale

Setting a shopping agent to think harder made it surer of its choice and rarely made the choice measurably better. In a University of Chicago preprint on ten frontier models, posted on 17 November, the agents also tended to fail together, often on the same wrong product. Two off-the-shelf fixes, more reasoning effort and a second model’s opinion, leave those errors in place.

Adi Krishnamoorthy and X.Y. Han gave each model a request and a fixed, shuffled list of ten products with a single right answer, from WebShop, Amazon’s SQID or Wayfair’s text-only WANDS. The models were OpenAI’s GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna; Anthropic’s Claude Fable 5.1 and Claude Opus 5.5; Google’s Gemini 4 Argon and Gemini 3.1 Pro; DeepSeek V4.1 Flash; Alibaba’s Qwen 3.8 Max; and Moonshot’s Kimi K3. Each ran at low, medium and high effort, where offered, rating every pick out of 10.

The design’s limits shape every number below. Fixed lists leave out search, paging and navigation, where the authors say effort might matter more. WebShop is scored by a program, but a hand check found 15 of 100 sampled distractors arguably acceptable (95% confidence interval, 9% to 23%). SQID and WANDS rest on crowd labels, whose noise may inflate apparent shared errors, since a mislabelled task counts every model wrong. Each condition ran once, and confidence was a single integer returned alongside the answer. The authors say tasks, logs and code will be released later.

Accuracy barely moved from low effort to high, against levels that ran across the ten models from 74.0% to 88.1% on WebShop, 43.7% to 50.6% on SQID and 65.8% to 73.7% on WANDS. Of thirty comparisons in Table 1, summarised below, a single low-to-high gain survived the Holm correction for multiple testing, which the authors ran over 26 contrasts per dataset: Qwen 3.8 Max on WebShop. Its gain came entirely at the step to medium, as did GPT-6 Luna’s on WANDS, which also survived correction there. The authors read such steps as low effort being “genuinely insufficient” for some tasks.

Low → high WebShop SQID WANDS
Success change, points -1.4 to +5.5 -2.0 to +2.0 -2.1 to +5.2
Confidence change, points -0.6 to +3.2 -0.8 to +2.5 -0.8 to +4.5
Models passing Holm, success then confidence 1, 8 0, 4 0, 5

Stated confidence moved in one direction. On WebShop it rose for nine of ten models, mostly from levels already too high.

Gemini 3.1 Pro shows the gap. High effort lifted its WebShop accuracy by 3.2 points, a gain that did not survive correction. Its WebShop AUROC, which measures how well confidence separates right answers from wrong ones, fell from 0.66 at low effort to 0.54 at high, where 0.5 is chance.

Nor did “high” mean the same work everywhere. OpenRouter, the gateway the authors used, bills reasoning tokens as output. At high effort, Qwen 3.8 Max spent nearly fifty times as many reasoning tokens per WebShop task as GPT-6.1 Sol. The same label thus buys very different amounts of billed thinking.

Calibration followed the model more than the setting. Anthropic’s Claude models were best calibrated overall on every benchmark by expected calibration error, though Gemini 4 Argon beat both on WebShop’s Brier score. The lead is relative: on SQID the two Claude models still overstated their success by 12.4 and 10.8 points, and on WebShop the other eight overstated it by up to 15.8.

The setting moved calibration in places. Qwen 3.8 Max’s WebShop calibration error fell from 12.0 to 6.7 points as effort rose, and Gemini 3.1 Pro’s AUROC fell as above. A calibrated agent, the paper says, “can hand the decision back to the user when it is unsure”. A cut-off that caught most of the overconfident models’ errors would, the authors say in a supplement, discard most of their right answers.

The authors offer two reasons effort bought confidence rather than accuracy. Choice may be “bounded by reading rather than reasoning”, or a long trace may give a model “the experience of having checked, which it then reports as confidence”. In the Paderborn tests we covered on 9 November, agents mostly kept a preselected product, reasoning or not.

A second opinion falters for a different reason: the models err together. On WebShop at low effort, 47 of 493 tasks beat every model, where independent errors would predict practically none.

Crowd-labelled benchmarks showed the same shape. At low effort, 185 of 500 SQID tasks beat every model, against almost none expected by chance. Label noise, the authors warn, may explain part of that cluster. On WANDS, 55 tasks did, and higher effort barely moved the error correlation.

Wrong answers also concentrated. Across tasks with multiple wrong picks at low effort, over 80% went to each task’s most popular wrong product, against 32% to 36% by chance. On WebShop the shared pick is typically “a plausible near miss” that fails “a secondary constraint such as size, scent or price”. SQID’s shared pick is most often a substitute whose title nearly matches the right product’s. The paper does not describe the shared pick on WANDS, and it leaves the cause open: “shared training data, shared listing cues, or both”.

No benchmark targets clothing, though WebShop includes Amazon fashion listings; the paper reports nothing by category. Its near miss is, on our reading, a clothing catalogue’s normal state: most styles in many sizes and colours under near-identical titles, much as SQID’s supplement describes substitutes that are often “the same product in a different size, color, pack count”. A near-miss listing that draws agents, the authors write, “captures demand across all agents at once”.

From the seller’s side, agents agreeing on the wrong size could look like a bestseller.

The strongest objection is that the paper tested the easy part and labelled it imperfectly. Ozan Candogan’s fashion test, which the paper cites, put three consumer agents (dots, Instinct and Muse) to 10 sweaters described by three attributes and then to 50 dresses described by 12, and found they “performed much more poorly in this richer setting”. He tested products rather than base models and did not vary effort, and he noted that search strategy “may shape the final choice as much as the model does”. The thesis fails if extra thinking pays off in search, which the authors say it might, or if most shared errors are label artefacts.

A payoff in search would shrink the thesis to the final pick, where every pipeline ends. There, in comparisons of each model with itself on identical ten-item lists and labels, the dial raised confidence without reliably raising accuracy. The label objection partly lands: the paper never checks how many shared errors fall on its arguably acceptable distractors. Re-scoring the 47 tasks that beat every model would show how many are the benchmark’s fault, once the data is released.

The choice, then, is what to pay for. Choosing a calibrated model, the authors conclude, “matters more than turning up its effort”; our own advice is to test calibration in-house rather than take a vendor’s word. If shopping agents read sales counts as agents in the Music Lab re-run we covered in October read a tally, a shared near miss could climb the rankings on votes that all came from the same mistake.