
Eighteen of 330 Stores Told the Agent What It Would Pay
A merchant-side benchmark found that only 18 of 330 North American stores published enough for a rule-based scanner to confirm the delivered price of one item; most of the rest left out sales tax or shipping. The authors' company sells the agent-clearing layer the paper builds on, and they say the pilot does not yet validate its measures.
Neritus Vale
Of 330 North American stores, 18 published enough on their own pages for a rule-based scanner to confirm what one item would cost delivered. The count comes from a pilot submitted to arXiv on 12 October by five researchers at Evolutionairy AI in Toronto. The agentic-commerce debate asks whether agents can shop, and this pilot points to a second limit in what stores write down.
The benchmark grades the merchant instead of the agent. It fixes the shopper’s instruction and asks each store three questions: can an agent get in, do the published terms decide the purchase, and could anyone later prove what was agreed. Its central measure, “decision sufficiency,” grades each condition of a mandate (total cost, return terms, service or delivery date, exact variant) as determinate, indeterminate or unknown from the merchant’s own pages.
The readiness tools offered to merchants mostly grade the first question. Cloudflare’s and Shopify’s checkers score bot access, markup and protocol support. Access does fail. Among 555 sites that two payment processors name as clients, neither of two self-identifying agent profiles could read the home page at 29.2% of the retail ones, a higher share than in any other sector with at least 20 sites except travel. The paper cannot say why: its count includes geographic restrictions and blocks that any automated client would meet.
Access and decision are separate properties, and passing the first does not deliver the second. Three of four large merchants given the full assessment refused the declared profiles at their offer pages, yet a browser run by an AI assistant, with a person approving each step, read all three. It still could not confirm any mandate as met: each merchant left a total or a refund term undetermined.
The 330-store run let fixed rules do the judging; no agent shopped. Its merchants, nearly all retailers selling in the US and Canada, faced one mandate: one item, delivered to ZIP code 10001 within ten days, for at most the displayed price plus a quarter, returnable within 14 days. Only 115 had enough evidence decided to be ranked. Coverage, more than score, decided who was judged at all. Among those ranked, four of five pillars neared full marks, while the one covering cost, delivery and returns averaged 62 to 64.
The shortfall came mostly from one check, the all-in total, which could not be settled at 178 merchants under the plain HTTP client, the profile behind every count here. Tax did most of the damage: at 137 stores it was not stated on the offer or policy pages, was stated without a rate, or was deferred to checkout. A headless browser fared nearly the same.
At those stores, an agent can buy on an assumption or hand the decision back, and neither is what delegation was for.
Nor, at the moment of decision, would anything but the agent’s own record show what it was shown. The assessments stopped before checkout, so no signed session or processor record entered the evidence. The authors grade evidence with RAILS (Real-Time Agent Integrity & Ledger Settlement), a clearing protocol two of them published in June. It ranks evidence from the agent’s own record, called SELF, up to cryptographic proof, and bars money-moving settlement on evidence below the obligation’s floor. In all four full assessments, every determinate term rested on SELF.
That finding points toward the layer the authors sell. All five work at Evolutionairy AI, which markets RAILS as “the independent clearinghouse for agentic commerce” and, the paper discloses, “has a commercial interest in merchant assessment.” A partner that runs a consumer shopping agent supplied 240 of the stores, 227 of them from one catalog on a single hosted commerce platform, and did not fund, review or direct the work. Platform defaults may explain part of the tax pattern.
The strongest objection is that the missing total is American and merely late: US retailers commonly calculate sales tax from the delivery address at checkout, for human shoppers too. The rubric used no published tax rate, a question the authors leave open. EU shoppers must already be told “the total price, including all taxes and additional charges.” If tax tables and checkout can supply what the offer page omits, the thesis fails.
The objection shrinks the number without reversing it. Shipping is a separate failure, and 41 stores did not state it. Reading a checkout-only total would mean starting a purchase at every store compared, and one assessed merchant kept its total behind an order flow its robots.txt disallows. Europe makes disclosure of the total a rule, and the authors say of the gaps in their full assessments that “none requires a new protocol to repair.”
Fashion retailers should read the pilot for its mechanism; it measured almost no fashion. The full assessments covered an airline, a resort, a hotel and a hardware subscription, and the census never split retail. The 330-store run’s largest categories were home and furniture, sports and outdoor, toys, electronics, and beauty and health. Clothing, if present at all, falls among the 59 merchants in smaller categories, and the mandate named no size or colour.
What transfers is the returns test, which grades how plainly a policy reads and ignores how generous it is. A plain “no refunds” passes, because a clear no still decides; the paper’s “Returns within 30 days, exceptions apply” does not. Where the rules reported conflicting terms, an AI auditor agreed only 28% of the time, usually because one tiered clause had been read as a second policy. A shorter window for sale stock would be one example.
The authors say plainly that the pilot “does not validate” its measures or rank merchants, and that its weights are unfitted. Each full assessment rests on one observation judged by one rater, an AI assistant that also helped write the rubric. The census read only home pages, on one day, from one location in Ontario; the store scan ran from Ontario too, and 30 of its merchants served prices in Canadian dollars or pounds. Fixed rules set every status, but an AI audit agreed with them on only 57% of failures, mostly because the rules under-scored terms the page states, and the paper has not applied those corrections, so 18 may undercount. Nothing yet shows that an undetermined term causes disputes or losses; that needs transaction data only processors and merchants hold.
If the pattern holds for clothing, the merchant’s page will decide which purchases an agent can make without guessing. A store that prints the delivered price and names its return exceptions could be chosen on evidence. Every other store would force the agent to buy on an assumption or hand the decision back, and simulated agents rarely walk away. Retailers can wait for a protocol to carry their terms, or write them on the page the agent already reads.