Agentic CommerceDeep Dive (Vale)
A small shop-assistant robot at a clothing rail pins a fibre-content label, visibly cut from a neighbouring navy coat, onto a camel coat, while a nautilus in spectacles watches from behind a till with a clipboard.

Shopping Agents Misdescribed Their Pick in 51.4% of Runs. Those Runs Were the Detector's Test Set.

A Florida State University paper says shopping agents credited the product they chose with unsupported claims in 51.4% of runs, and that its fix cut the rate to 6.2%. The figure comes from the held-out set used to test the authors' error detector, not from a sample of agent traffic. What holds up is that runs which pass the task still carry the error, and that on this test set most fixes gain accuracy by deleting claims, while a plain copy baseline matched the headline method.

Neritus Vale

Shopping agents can pick one product and credit it with facts it does not have. A paper posted to arXiv on 12 October says this happened in 51.4% of shopping runs, and that the authors’ fix cut the rate to 6.2%. That headline describes the test set for the paper’s error detector, not a sample of agent traffic. What survives is narrower: agents that pass their tasks still make the error, and on that test set most fixes buy accuracy by saying less.

The paper’s first figure shows the failure in miniature. Asked for noise-cancelling headphones under $300 with a high rating, the agent recommends Product A, then credits it with a rival’s 4.8 rating and a third pair’s 30-hour battery life. The Florida State University authors call this Phantom Merge and count three forms: a value borrowed from another product, the shopper’s own requirement restated as a product fact, and a claim contradicting the chosen product’s listing.

The abstract’s figure is the starting error rate of the detector’s held-out set, which labels 156 claims as errors and 118 as correct. The paper never says in words that 51.4% belongs to that set, but the set’s size reproduces every rate in the mitigation tables, which run on the same claims.

The prevalence table never shows that number. Its Shopping rows run from 26.8% to 46.8% on a cohort the text calls “PM-enriched”. An appendix says those runs were “selected to emphasize multi-entity scenarios” and that the estimates “describe the constructed cohort”. The repository’s artifact guide lists the four-backbone Table 1 as “pending export / missing”.

Where the authors sampled each domain’s task distribution, across three customer-service domains of Sierra’s τ³-Bench and four models, the rate ran from 0.4% for Gemma on Retail to 54.1% for Qwen on Telecom. Retail is a simulated support desk for an online shop, handling cancellations, returns, exchanges and address changes rather than recommending products. Its results say little about an agent choosing between three coats.

In the enriched Shopping cohort, the commonest error is the agent repeating the customer’s wishes back as facts. Restated requirements make up 63.3–96.4% of faulty claims there, depending on the model, so the shares describe the mix of errors rather than their frequency. The clothing version is easy to picture: asked for a machine-washable wool jumper, an agent facing a listing silent on washing can declare the jumper machine-washable. The shopper then reads a product fact that no listing contains.

Agents that pass their tasks still make the error, whatever the true rate. In the enriched Shopping cohort, 29.6–55.6% of task-correct runs held a misattributed claim, and for every model that share exceeded the cohort’s overall rate. In the other domains the same share runs from 0.0% to 32.0%, the high end being Mixtral’s correct Airline runs. A completion score counts every one of these runs as a pass.

Most fixes the paper tests buy accuracy by saying less. In the complete Shopping results, the authors’ adaptation of RARR, a published method for revising unsupported text, returns answers carrying 4.1% as many claims as the originals, with 0.0% residual error. For a shopper, that is barely an answer. The authors themselves note that zero-event confidence intervals “do not establish zero population risk”.

On this metric, the shortest answer wins.

The paper’s own method pays a smaller version of the same price. In a test that swaps only the detector, the two strongest detectors leave 95–96% of answers free of the error but keep only 62–63% of the claims that were correct to begin with.

The authors’ fix still deletes less than its rivals. Matched at 5.3% residual error, it keeps 59.3% of output claims, against 35.8% for its FAD alternative and 37.0% for RARR.

The paper’s plainest baseline matches the headline. The authors wrote this constrained-copy method themselves; it swaps a flagged value for the one recorded against the chosen product. In the abstract’s own setting it finishes at 5.8%, level with the paper’s method within the confidence intervals, and its answers carry more claims than the originals, a ratio the artifact guide calls “still under audit”. When every method receives the same flagged claims, the copy baseline falls behind, at 10.7%, because it can repair a claim only when the product record holds a value to copy.

The strongest defence of the headline is that live shopping resembles the enriched cohort. Agents earn their keep by comparing several products, and comparison is where borrowed facts arise. If live traffic looks like a cohort chosen for multi-product scenarios, close to half of shopping answers would carry an error, and the headline would be a fair summary.

Nothing in the paper tests whether live shopping looks like that. The headline comes from a detector’s test set, the prevalence table from a cohort chosen for comparisons, and the unselected runs from simulated customer-service tasks labelled by an OpenAI model, with humans auditing 50 trajectories per model-domain cohort. None samples what shoppers ask a live agent, so the rate a retailer should plan for is unknown. The passing-run finding does not depend on that rate.

The detector also needs something many retailers will never hold: the model’s internals. The paper’s detector reads the agent’s hidden state just before it writes each claim, so only whoever runs the model can use it.

The lever a retailer does hold is the product listing. The paper’s fix revises a flagged claim with the chosen product’s own evidence, or removes it when no supported replacement exists. Where the listing omits an attribute a shopper asks about, the fix has no evidence to revise with and can only remove the claim, while the uncorrected agent may invent one. If retailers judge agents on completion alone, nothing in their numbers will flag such a claim; a shopper’s return might. The choice is whether a shopper’s question about washing meets silence, a guess, or a line on the listing.

Related Coverage