Research & TrendsDeep Dive (Vale)
A row of wooden mannequin shoppers faces a giant phone showing two versions of a clothing feed marked A and B; most barely lean toward the screen, one leans close to a chalk line, two topple far past it.

Eight Simulated Shoppers Replayed a Live A/B Test. Only the Authors' Own Came Within 30%.

UserArena replayed one production A/B test of a ranking change through eight simulators. The authors' own method registered 71% of the live effect, the trained rivals under 30%, and two GPT-6 models overstated it. The check covers one click metric, with no sales data and no error bars.

Neritus Vale

A simulated shopper can score well against logged clicks and still misjudge what a ranking change does to the people it imitates. UserArena, a benchmark posted on 22 October, shows it by replaying one live A/B test through eight simulators. Their estimates of the effect ran from almost nothing to more than twice the real one. Only the method the authors propose came within 30% of it.

Recommendation teams want simulated shoppers to stand in for live tests, which are slow and can show real customers worse rankings. Meta researchers have proposed RecoWorld, a blueprint for simulated environments where agentic recommenders “can learn from errors without impacting real users”. A KDD 2026 paper from City University of Hong Kong and Huawei said LLM agents show “great potential to replace traditional online A/B testing”. Its yardstick for “real-world performance” was ranking accuracy on held-out movie and Amazon Fashion logs, not a live test.

UserArena is built from 64,729 recorded sessions on what the paper calls “one of Asia’s largest e-commerce platforms”, left unnamed. Its authors work at Leiden University, Renmin University of China, the Chinese Academy of Sciences and Alibaba Group, which supported the work through its Innovative Research Program. They rebuilt the platform’s two-column recommendation feed and product pages, so a simulator can scroll, open items, add to cart and buy. Apparel drew 54.90% of the logged item clicks.

The decisive check replays one production A/B test through each simulator. The authors ran the test’s pair of ranking policies on a frozen pool of 10,000 simulated users, split evenly between the arms. They then set the simulated gap in page-view click-through rate against the gap recorded online. Their ratio is the score: 1.0 means the simulator reacted exactly as real shoppers did.

OPTD, the training method the paper introduces, registered 71% of the live effect. Four trained rivals, STA, Customer-R1, Shop-R1 and AlignUSER, run on the same small Qwen model, which holds model size fixed while prompting, memory, observation mode and post-training vary. The best of them, STA, registered under 30%.

The weakest simulators saw almost none of the change. AlignUSER, which Bougie and colleagues built to evaluate recommender systems, registered 2%. The untuned Qwen model registered 0.4%. A team relying on either would have read the new ranking as doing next to nothing.

OpenAI’s general-purpose models erred the other way. GPT-6 Sol, listed under “More models” as built for “complex coding and agentic workflows”, put the effect at 1.576 times the real one. The listed flagship, Astra, was not tested. GPT-6 Luna, which OpenAI calls its “most efficient model”, more than doubled the effect.

On Table 3’s own error column, GPT-6 Sol comes second, which hides a difference in kind. Its overshoot of 58% is a smaller miss than the undershoot of every trained simulator except OPTD. The two failures mislead in opposite directions: one inflates a change, the other shrinks it, at worst to almost nothing.

Matching logged behaviour is a different skill from reproducing a change. In the paper’s single-session test, Customer-R1’s click-rate error of 0.0227 was the second smallest, behind only OPTD’s. Replaying the live test, it registered 19% of the effect. The two tables use different click metrics, clicks per feed action in one and per page view in the other, so the figures are not interchangeable; a low error against the logs did not carry over to the A/B replay.

The validation rests on that one test, and the appendix leaves out most of its particulars. It gives no duration, no traffic volume and no size for the recorded effect. Nor does it say what changed between the rankings, beyond which items were shown. With no confidence interval on either side, the paper cannot show that AlignUSER’s reading differs from zero.

The test also measures clicks, not sales, and outsiders cannot rerun it from the release. The paper’s cart-rate and terminal-outcome metrics compare simulated sessions with logged ones, never with the live test. The project page captions its example trajectories with the caution that a logged “Buy” is “not confirmation of payment”. The public repository holds 535 held-out sessions and the main table’s scoring code, while the project page lists the full dataset as “Release planned”.

A separate Turing-style test asked whether single trajectories look human, and OPTD’s were the hardest of four to spot. Annotators with recommender-system or annotation experience compared real and simulated trajectories side by side, blind to which was which. When they committed to a guess on OPTD’s pairs, they named the machine in 59.5% of cases, by our arithmetic from the paper’s 38.7% and 26.3% win rates. Looking like one shopper is a separate test from responding like a crowd of them.

The strongest objection is that a ranking team needs only the direction of an effect, and every simulator here got it. If each simulator shrinks or inflates every change by its own fixed factor, the order of candidate rankings survives, and STA would pick winners as reliably as OPTD despite its lower score. For the thesis to fail, that fixed factor has to exist.

One test cannot show that such a factor exists, because a constant shrink and one that varies with the kind of change look identical on a single pair. Training raises the stakes. If a recommender is tuned against a simulator that registers a fifth of each change, it has little signal to climb. Inflation does different damage, rewarding the recommender for chasing whatever the simulator overreacts to.

OPTD is the authors’ method, scored on the authors’ protocol, against one test that only they can rerun.

Any platform with an archive of past A/B tests already holds the raw material UserArena used. Replaying a handful of them through a simulator, before tuning anything on it, puts no customer in front of a worse ranking. Skipping that step means letting an uncalibrated model of shoppers decide which rankings real ones get to see.