
Show Research Agents a Photo and the Shopper's History Counts for Less
A new benchmark finds that adding a photo to a shopping request improves research agents' overall ranking but weakens how closely they follow the shopper's own ratings, and the three commercial models tested did not fix it. Walmart's case for photo search in Sparky rests on a conversion figure from an executive's September remarks, which cannot show the difference.
Neritus Vale
Walmart and Amazon want shoppers to show their AI assistants a photo, and a new benchmark finds that in research agents the photo crowds out the shopper’s history. Walmart’s holiday post of 30 October invites Sparky users to “recreate the look using an inspo pic”. In PMRecBench, posted on 4 November, photo-and-text requests improved agents’ overall ranking but weakened their “preference following”: ranking by what a user’s own record says they like.
The paper tested no retailer’s assistant, Sparky included; its agents are research designs such as ReAct and RecMind, run on open and commercial models. Its authors are Xiaoyu Zhang, Bin Wu, Sahan Bulathwela and Emine Yilmaz of University College London, and Yougang Lyu of the University of Amsterdam. They cite Amazon’s Alexa for Shopping as a live interface for image-text queries; its launch page lists “snapping a photo to use visual search” among tools shoppers already use, and “shopping history” among its personalisation inputs.
The clothing half pairs real shoppers’ ratings with requests a model wrote. Its 198 users come from a research dataset of Amazon clothing reviews; the film half uses MovieLens ratings. Gemini 3.1 Pro wrote their 258 requests from attributes recurring in each user’s likes or dislikes, attaching photos of items that user had rated. Claude Opus 4.8 and GPT-5.5 judged the wording, and the authors checked disputed cases. Because the photos show items the user had already rated, the paper did not test outside inspiration photos, and the Walmart scenario extends its result rather than repeating it.
Each request hides its target, an item the user later rated highly, among two groups of rivals. The paper gives “rated highly” no number; a like is a rating of 4 or more, a dislike 2 or less. One group holds favourites that break the request; the other, items that meet it but that the user disliked or never interacted with. Only the history separates the target from the second group, so beating it is the paper’s proxy for preference following. For most queries that group is partly items the user never interacted with, so some rivals may be things the shopper has simply never seen.
The agents handled the first group and stumbled on the second. On clothing, four agents on the open-weight Gemma-4-31B ranked the target above the request-breaking favourites in 93% to 95% of pairs. Reading the request was the easy half. Against items that matched the request but not the shopper, they managed 71% to 76%.
Adding the request costs preference following. On clothing, the score fell by 4.6 points, a mean over four Gemma-4-31B agents with a 95% confidence interval of −8.8 to −0.5 points, while overall ranking rose. The direction held for each agent in both domains on both backbones; the paper tests the drop as a mean, not agent by agent. Qwen3.6-35B, reported in an appendix, fell 5.6 points on clothing.
The agents also began from a modest base. On clothing, the paper’s history-only collaborative filtering scores 0.70 and simple score fusion 0.72, and the authors do not find agents consistently ahead of fusion.
The authors trace the loss to how agents read the picture. When a plain-text request named the attribute instead of showing it, preference following no longer fell significantly. Having the agent’s own backbone describe the photos in text made the fall steeper on movies, with the same pattern on clothing. Agents, they write, “can prioritize resemblance to reference images over evidence from user histories”.
The authors suggest one reason history helped less on clothing: agents were “rarely inspecting clothing history items”. Their appendix on tool use does not support a simple story of neglect, though. Tool calls rose in 14 of 16 settings, and the authors write that the decline in preference following “is not caused by insufficient tool use”. In one traced case, an agent on GLM-5.3-Flash noticed the shopper’s history was mostly menswear, yet still put two women’s items first and second, for a black satin look the request never asked for, and the men’s target third.
The three commercial models tested did not fix it. ReAct, a general-purpose agent, ran on eight backbones: small open-weight Qwen3-VL-8B, Ministral 3 8B and Gemma-4-E4B; medium open-weight Gemma-4-31B and Qwen3.6-35B; and commercial GPT-6 Luna, GLM-5.3-Flash and Claude Haiku 5.5. On clothing, the commercial models read requests about as well as Gemma-4-31B, yet all fell below its 0.76 for following the shopper, scoring 0.68 to 0.72. Qwen3.6-35B, at 0.72, sits at the top of that range, so the gap is with Gemma, not with open-weight models generally. The abstract concludes that “stronger commercial backbones alone fail to resolve this imbalance”; its “stronger” covers these three models.
The failure traces agree. Claude Opus 5.5 labelled 125 sampled failures, 25 per model, checked against GPT-5.6 Sol and human labels. In 35.2% of them, the agent read the request correctly, then picked an item that looked closer to the reference photo. None came from ranking on the history alone.
Walmart’s conversion figure measures something else. The 30 October post carries no figure; the number comes from a September investor-conference remark, which Retail Dive repeated on 5 November: visual search “led to” a 57% rise in conversion over text-based searches, according to Walmart U.S. chief executive David Guggina. The transcript of his 15 September Goldman Sachs appearance is narrower: “we see the conversion rate jump by 57% versus text-based shopping.” That compares two kinds of search, much as the Sparky basket figure we examined in October compares two kinds of shopper.
Conversion, like the paper’s overall ranking score, nets both effects, so it can rise while personalisation slips. Guggina’s own example was a co-worker’s photo of his lawn, from which Sparky, “just with that context”, suggested seed and fertiliser (transcript). Lawns need little history. Copying a look from an inspo pic is different: only the record says who will wear it.
The strongest objection is that a photo should outrank the past. A shopper uploading an inspo pic may want to dress differently, and an agent clinging to old ratings would fail them. For the thesis to fail, photo requests would have to contradict shoppers’ histories so often that following history becomes the mistake.
PMRecBench was built so that no such contradiction arises, and the agents still drifted. Requests come from the user’s own rating patterns, “so that the query introduces no requirement that conflicts with this user’s preferences” (introduction). With no new taste to follow, the drift cannot be an agent updating to one. Against rivals the user had disliked, adding the photo cut clothing preference following to 0.517 on both medium backbones, close to a coin toss. The authors call that subset’s coverage limited: it covers 14.70% of queries.
A photo can tell an agent what to match; only the history can tell it whom the match is for.
On the August earnings call, chief executive John Furner told of a meal plan for which Sparky “recognized the ingredients they had recently purchased”. That shows Sparky uses purchase history; it does not show photo results do. If Walmart keeps scoring photo search on conversion against text, a photo that sells and one that serves the shopper will look alike in its numbers. Telling them apart needs data Walmart holds: whether photo results favour the brands and sizes long-time shoppers keep over those they send back. Running that check is a choice, and so is quoting conversion instead.