Agentic CommerceDeep Dive (Vale)
A brass robot shop assistant offers a basket of cat litter and cat food to a puzzled man holding a financial newspaper, beside an index card altered from 'Finance / Stocks' to 'Pet / Cat' and a dusty, unrung bell marked 'ERROR'.

For AI Agents, a Broken Tool Beats a Wrong One

RITE, a benchmark from HKUST and Tencent researchers, finds that a plausible wrong value from a tool cuts e-commerce agents' success by about half or more, does more harm than an outright failure, and mostly goes unnoticed. The benchmark tested a fix inside the agent; the tool-side repair, errors that announce themselves and answers that carry their age, is our inference.

Neritus Vale

An AI agent copes better with a tool that breaks than with one that is quietly wrong. RITE, a benchmark posted on 20 October by researchers at HKUST and Tencent, fed ten models a plausible but incorrect value from one tool, or occasionally a tool argument. Success fell by about half or more, and every model did better when the same step failed visibly. For retailers wiring agents into stock and price systems, the tool-side repair is our inference; RITE tested only a fix inside the agent.

In one case study, a user’s reading history held an article filed under Finance, Stocks. In the copy GLM-5.2 received, the record said Pet, Cat, and nothing marked the change. The model made that record its main evidence, read the lack of finance videos as support, and asked for keywords. They came back as canned cat food and cat litter.

The workflows are mostly live-stream tasks. Inputs from one internal platform’s production logs ran through workflows deployed on another, and the paper names neither; none of its nine workflows is a stock, pricing or ordering system.

Each test hands an agent a recorded run up to one step, where the value is correct, silently wrong or visibly failed. The 167 slots draw their wrong values from a taxonomy of seven types, among them a sibling category, a number nudged into the next range and a stale value. Staleness was defined but never run, because recognising it “requires prior or time-indexed evidence that the agent does not observe”. An LLM judge scores success and is stricter than human raters, who marked 89 of 100 rollouts solved against its 69; read the gaps, not the absolutes.

Silent errors cost even the most resilient model nearly half its success. Claude-Opus-4.8 succeeded on 76.5% of tasks with correct values and 40.3% with silently wrong ones. A visible failure at the same step hurt it less, though not by enough to restore its clean-run score.

Models flagged and acted on a silent error in only 20.8% to 33.6% of runs. In 99.6% of the rest, the latest tool output for that value was still the corrupted one.

Noticing was not the same as repairing. Among runs where a model did notice, 57.8% still held the corrupted value. Qwen3.6-35B restored the right time window, then swapped in one of its own.

Scores on correct tool values put the two most resilient models near the bottom. Claude-Opus-4.8 and GPT-5.5, the closed models, came ninth and tenth by thin margins when every value was right, and first and second when one was silently wrong. Choosing an agent model on clean-run scores means reading the wrong column.

On the paper’s own numbers, a visible failure usually mattered more than the choice of model.

Pooled across the ten, replacing a silent error with an explicit failure raised success on the same slots by 21.4 points. The strongest model led the weakest by 12.6 points under silent errors, our arithmetic from Table 1 and on a different basis. The gain was uneven, and Opus’s was the smallest.

The strongest objection is that this is a model problem, and better models will solve it. The closed models kept 51.7% of their clean-run success against 39.1% for the open-weight eight, so scepticism may keep rising until agents doubt a plausible value unaided. The thesis fails if a model can tell a wrong value from a right one using only the value and its context.

The paper’s nearest test of that condition points the other way. Its State Checker told three models, on 81 slots, to treat every tool result as possibly unreliable. Noticing rose by 12 to 38 points; success rose by at most nine, and Claude-Opus-4.8’s not at all. “Making agents look… is therefore easier than making them repair what they find,” the authors write.

The Model Context Protocol, behind the Otto plugin we covered earlier today, suits the failure that announces itself. Its specification has tools report errors as “actionable feedback that language models can use to self-correct”. An output schema checks an answer’s shape, which a wrong but well-formed stock count passes. Freshness hints are required on tool lists and resource reads, not on a tool call’s result.

That leaves a stale tool answer without a required label, and stock and price data go stale by sitting still. RITE never ran a stale case, so the time-indexed evidence its authors say detection would need is untested; the specification lets a tool result carry a modification time only as an option. The benchmark is also generous: every instance is recoverable by design, since a re-query returns the clean value. Persistent faults, the authors concede, “may invalidate this path”.

The error type nearest to a stock count or a price is a number nudged into the next range. It kept only 29.7% of clean-run success and drew the highest noticing rate, 34.2%. Category errors did more damage: swapping in a broader or narrower category left 14.0% of tasks solved, across only 17 slots. The authors suggest that a wrong number gets checked against the tool, while a wrong category “appears anomalous only in context”.

RetailBridge, a KAIST-led benchmark posted on 18 October, shows the opposite failure. With noise seeded into its records, an agent’s accuracy on a 200-question subset fell from 0.410 to 0.230, “mostly because the agent treats corrupted names and categories as exact keys and misses records”. There the agent missed a record it should have found; in RITE it accepted a wrong one.

RITE measured agents, and the tool-side repair argued here was never run. A retailer can buy the most sceptical model and, on this evidence, still lose about half its success to quietly wrong values. Or it can make every price and stock answer state its age, and let a tool that detects its own fault say so. Either way the price is the one the cat-food case shows: a confident recommendation built on a record nobody flagged.