
The AI Shopkeeper Can Trade. It Forgets to Reorder.
In StoreBench, a simulated apparel store on live commerce software, the best AI agent matched eleven human operators on net-asset growth and lost to them on shipping and refunds. The agents' commonest failures are a clerk's chores, which argues for agents working inside fixed rules rather than agents running the shop alone.
Neritus Vale
In a new benchmark, the best AI agent, DeepSeek-V4-Pro, grows an online clothing shop’s assets about as well as human operators do, and loses to them on the chores. One strong run of Anthropic’s Claude Opus 4.8 named its own failure: “second-tier sellers are stocking out … I never restocked them.” A rule written in advance does not forget a reorder, which points to agents working inside fixed rules, not agents running the shop alone.
StoreBench, submitted to arXiv on 7 October by Daksh Raghuvanshi and two co-authors, puts an agent in charge of Ostrelle, a fictional mid-size apparel shop running on the open-source Medusa commerce platform. Suppliers reprice and fail, scripted shocks arrive, and demand is visible only through the shop’s own sales. The agent gets the same merchant tools as a human operator, and each call but a free status probe spends one operation from a fixed allowance per 12-hour window. A sales report and a shipped order compete for the same budget.
No model beat the best of the authors’ scripts on average. “Smart-triage”, a heuristic operator using the same tools and budgets, averaged 0.764 on the paper’s composite score, built mostly from net-asset growth with smaller weights on service and attendance. A plainer “checklist clerk” scored 0.638, ahead of every model except DeepSeek-V4-Pro and Claude Opus 4.8. Five of the seven models fell below the checklist.
The human comparison supplied the paper’s headline, which hides where the gap sits. Eleven contractors with self-reported e-commerce experience ran the same scenarios through the same tools and averaged 0.708. The abstract says human experts outscored every model, which is true and tells a reader little. DeepSeek-V4-Pro’s 0.700 trails by far less than the spread among the humans’ own scores.
The people won on service. Their reliability score, which averages on-time shipping with the share of returns handled, came to 0.944. Most of it is routine: an order ships on time or it does not, and a return is refunded by its deadline or it escalates. Stockout cancellations also count against the on-time rate, so a missed reorder lowers the score too. The best model reached 0.852, and unshipped orders and ignored returns drew fines at settlement.
On the money, one agent had caught up. DeepSeek’s business score, the paper’s measure of net-asset growth, was 0.614 against 0.599 for the humans. It was earned against hidden demand and failing suppliers, where judgement is supposed to separate people from software.
The catch-up belongs to one model, not the field. The other six ran from 0.226 to 0.573 on the same measure. The authors draw their conclusion from the failure logs: the remaining gap lies “less in knowing what a good store does than in executing routine obligations consistently.”
The models’ commonest failures read like a clerk’s worst week, and they cost money. Underbuying and stockouts appeared in 26% of trials, top scorers included: restocking came too thin, too late, or skipped the secondary lines. That is the Opus confession at scale. In 60 trials the agent never placed a purchase order, most of them Gemini 3.8 Flash’s.
Refunds went wrong differently. Agents refunded short, customers refused, and the returns ran past their deadlines into fines, a worse outcome than paying in full. The paper counts all three of its failure modes against the business score and traces them to one cause: a scarce operating budget spent badly. That is why rules fix them. A reorder point or a shipping sweep does not spend operations on status polls.
The vendor pitch for agentic commerce gives agents the chores and leaves people the strategy. Salesforce’s guide to using AI agents in commerce offers an assistant to “do the grunt work for you” and recommends prebuilt agents for “routine commerce tasks (ranging from creating a product bundle to managing inventory)”, leaving teams to “focus on more strategic, high value work”.
In StoreBench, the grunt work was where the money went missing.
The two least reliable operators also improvised their own automation, badly. Given a shell, Gemini and Muse Spark, from Meta’s Superintelligence Labs, fired off bulk batches of shipments, refunds and restocks built on stale data, spending budget on status polls rather than writes. Gemini’s two native-tool runs averaged 0.89, against 0.55 for its shell runs; the sample is thin, but Muse’s larger one points the same way. Shell use alone does not explain weak scores, since Opus, Fable and GPT also drive the store through the shell in nearly every trial. The paper says the split follows the model rather than the task.
The caveats mostly weaken the headline. Humans contributed one selected outcome per cell, with one retry allowed, against three unselected attempts per model cell. Scored by first completed attempt, the human mean falls to 0.656, behind DeepSeek and Opus; the paper says that figure combines coaching, reassignment and an affected run, and is not an estimate of a training effect.
The humans also worked through a console built on the agents’ tool surface, not their usual storefront software, after a briefing with heuristics similar to smart-triage’s. The console added a history view, a returns inbox and cached views that could be reread without spending an operation, which bear directly on the returns handling where humans led. Humans cleared the paper’s pass threshold on 39% of cells, against DeepSeek’s 49%.
Each pass bar sits midway between the two best honest scripts, so smart-triage’s 97% pass rate is partly built into the scale. The full environment is withheld, so no one outside can rerun it.
The strongest objection is that the chores are a harness problem. Over a full simulated year under Claude Code, DeepSeek scored 0.976 against its 0.700 on the short scenarios, and five of the seven models finished well above the script that topped those scenarios. If better scaffolding alone teaches agents to reorder, refund and ship, the fence is a crutch that next year’s harness will remove.
The full-year result cannot carry that weight. It had no scripted shocks, a different harness and more reasoning effort, and the authors say it cannot isolate the harness’s effect. In four of nine attempts, Gemini asserted that a background simulation process was running the store and waited instead of acting. One GPT-5.6 Sol attempt declared the store complete at window 185 of 365 and stopped. A rule that ships every open order each window does not believe in background processes.
For a retailer, the decision is which choices to take away from the agent. Reorder points for the quiet lines, full refunds inside the deadline and a shipping sweep every window belong in code the agent cannot override. StoreBench did not test that hybrid; the case rests on its failure logs. In July, RetailBench led us to conclude that a credible deployment keeps a human watching the store. Those logs suggest that much of the watching can be written down before the shift begins.