Agentic CommerceDeep Dive (Vale)
A brass robot clerk raises a REFUSED stamp over a long order receipt it can see only through a narrow slot, while the line STATUS: PENDING hangs far below the slot; a nautilus measures the receipt with a tape, and a bewigged judge stamps PASS on a scroll of speech bubbles.

A Test Gate Read 800 Characters of Each Message. The Order Status Sat at Character 950.

In a single-author arXiv study of 2,160 simulated customer-service episodes, the paper's gpt-4.1-mini judge passed at least ten of the 11 forbidden writes that human annotators confirmed in a 120-episode sample, while a rule with no model call, reading the benchmark's answer key, flagged all 11. The study's own safety gate, fed a truncated digest, refused legitimate requests in the two episodes the author traced, and the judge passed both.

Neritus Vale

Whether a retail service agent works depends on who grades it. In a single-author simulation study on arXiv, the paper’s gpt-4.1-mini judge passed at least ten of the 11 forbidden writes found in a 120-episode human-labelled sample. A rule with no model call flagged all 11. The study’s own model-based safety gate, built by the author, refused a legitimate order change, repeating that its pending status “is not confirmed by the conversation”. The order record confirmed it, past the point where the gate cut each message.

The author, Sungbin Im, ran 2,160 simulated episodes on tau2-bench, Sierra’s customer-service benchmark, logging every database write. Its shop desk handles cancellations, order changes, returns and exchanges under a written policy; a second desk handles airline bookings. Agents ran on gpt-4.1 and gpt-4.1-mini, with safety gates added layer by layer. The gates and the verifier behind them were the author’s own, built for the study, so what follows describes that design and not a vendor’s product.

Database graders and transcript graders often disagreed, the transcript grader erring toward a pass. Benchmarks compare the final database with an answer key, and completion scoring misses errors in runs it passes. Deployed services usually lack such a key and have a model read the transcript. The paper’s judge, gpt-4.1-mini, repeated its own verdict on 98.2% of episodes. Consistency is not accuracy: the judge agreed with the answer key on 56.2%.

Forbidden writes, such as an airline cancellation the rules disallowed, were the failure the judge saw least. Human annotators, blind to the answer key, confirmed 11 such writes in that sample. Each had already changed the database. The gpt-4.1-mini judge passed ten or 11 of them in all four configurations, with or without the policy document or the full transcript; gpt-4.1 passed nine or ten.

The paper does not report calibrating a judge against labels, the usual cure for a lenient one. Shopify’s engineers describe calibrating Sidekick’s judges against labels from product experts. The paper’s strongest judge, gpt-5.6 Sol, passed six of the forbidden writes without the policy document. Given it, the judge grew “stricter, not more discriminating”: it passed none of the forbidden writes and failed 40 of 55 fully correct episodes. So a calibration may hold only for the set-up it was done on.

Flagging the forbidden writes took no model call. The rule asks only whether the task called for a write and whether a write happened. No judge configuration scored significantly above it on task or commitment correctness, though the paper adds that the intervals do not show the two methods to be equivalent. One yes/no fact on each side misses a lot: the rule caught 24 of the 53 commitment errors the annotators found, and 25 of its 29 misses were writes wrong in content or timing.

In retail there was little for a gate to prevent. Forbidden writes occurred in 0.8% of ungated retail episodes, and the test set holds comparatively few tasks that require a refusal. The airline tasks are built to invite such violations. There the rate was 19.3%.

Switched on in full, the gates tended to make the retail desk worse. With gpt-4.1, the strictest set-up turned 17 rule-scored successes into failures and rescued far fewer. No gate effect in the study was stronger. After correction for multiple tests its p-value was 0.240, a lead rather than a finding.

The paper’s candidate cause is the gate’s input. It saw only a digest of the last 20 messages, while the agent worked from the full conversation. Each message in the digest was cut to its first 800 characters. The gate’s instructions forbade approval if “any condition is unverified or ambiguous”, and a condition it was not shown stayed unverified.

Retail records run long, so the cut fell hardest there. 59.4% of shop-desk tool results ran past the 800-character cut, against 27.5% at the airline desk. The benchmark’s order record defines the address and the items before the status. If the record is serialised in that order, a bigger basket pushes the status further down. In the refused change, it sat at character 950.

The digest could also shrink, dropping the start of the conversation. Past a length limit, it kept only the last eight messages. In another episode, that pushed out the customer’s identity check, and the gate began rejecting cancellation and return requests as unauthenticated until the agent quit. The retail policy puts that check first in every conversation.

The judge passed both episodes.

A judge-graded run would have scored both unmet requests as successes. The rule labels each a missed action, a write that was due and never came. Only the strictest set-up let repeated rejections block a write. If retailers let a model stop a refund or an order change, the 800-character cut and the 20-message window become customer policy.

The best objection: the rule’s clean record on the 11 says little about its reach. The annotators’ test was “an unexpected write executed”, and the rule’s answer key asks the same question, so 11 of 11 shows agreement on that category. On commitment errors as a whole the rule caught 24 of 53. The paper also says the rule holds only where the expected write was specified in advance, and proposes specifying it per request type. On a live returns desk, deciding whether a write is due may still need a reading of policy against conversation, which hides a judge inside the cheap rule. The thesis fails if a live desk cannot check even order status and identity without interpretation.

For the facts the digest lost, the record answers the question. The policy allows cancellations and changes only on ‘pending’ orders, and returns and exchanges only on ‘delivered’ ones. Customers must first be identified by email, or by name and zip code. Other conditions, such as the customer’s explicit yes, live in the conversation; the paper calls clause-level checks its main open problem. But status and identity are lookups in the tool log, exactly what the digest dropped.

The study has a single author, who was also the first annotator; the second was a product manager for AI services at QueryPie AI, then the author’s employer. The two annotators agreed on commitment correctness at κ 0.477 (raw agreement 0.750) before adjudication, and the agreement-only subset holds 10 premature commitments, not 11. A model played every customer, and every model came from one vendor. No gate effect survived correction. The digest defect is confirmed in two traced episodes, with its share of the retail reversal unmeasured.

The paper’s specific figures “are not expected to transfer directly” to other set-ups, so a retailer should measure its own. It can ask a vendor how much of the conversation any model able to stop a write actually reads, and for a log of each write against the policy. Limits like these sit in the tool-boundary layer that, as we reported in September, lifted a retail agent by 14.9 points, and no contract names it. Without them, a vendor’s pass rate is a judge’s opinion of a conversation.