The scarce input in retail AI is no longer the model. It is the customer whose order you spend proving that one model beats another. Ranking engines, pricing rules and conversational shopping agents have converged on the same object: a swappable policy, cheap to produce and expensive to adjudicate. The supply of candidates has risen fast; the traffic available to choose between them has not moved. The binding constraint has shifted from what a retailer can build to what it can afford to compare.

Candidate policies are now written by other models, which is why there are suddenly so many. MetaStrategy, deployed on Taobao’s homepage feed, has an LLM emit a typed JSON bundle of objective weights, category preferences and position rules, then compiles it into a generator that “competes atomically with incumbents”. Netflix’s GenRec swaps a feature-engineered ranker for an LLM post-trained on ranking labels. Either can spawn plausible variants faster than a team can test them. Producing a challenger is nearly free; establishing that it is better is paid for in purchases.

The number of purchases a comparison needs is set by the effect you are chasing, and it depends on the baseline more than intuition suggests. Statistical power scales the required sample with variance over the square of the effect, a relationship Ronny Kohavi’s KDD 2015 keynote sets out as n proportional to sigma-squared over delta-squared. For a fixed relative lift, that formula rewards a high-converting site and punishes a low-converting one: the same percentage move produces a smaller absolute shift on a smaller base, which takes a larger sample to detect. At 95% confidence and 80% power, catching a one percent relative move takes roughly 7.7 million orders across both arms against a four percent baseline conversion rate. Against a one percent baseline, the same test needs about four times as many: roughly 31.7 million. Sensitivity is bought in transactions, and transactions are the one input a retailer cannot manufacture.

Adding candidates raises that bill twice, and the second increase is the one teams forget. A shared control helps, since the standard allocation against a single baseline sizes the control at the square root of the challenger count times each challenger’s traffic. Correcting for multiple comparisons pushes back, because a fixed family-wise error rate forces every test to a stricter threshold and larger sample. Net the two and the arithmetic is unforgiving. Three candidate policies cost roughly two and a half times a single A/B test. Fifty cost about thirty-five times, a multiple held down only because the shared control is doing real work.

Put those multiples against a real order book and the constraint stops being abstract. ASOS shipped 57.3 million orders in the 52 weeks to 31 August 2025, its results announcement shows. That is roughly 157,000 a day, which sounds ample until it has to be divided into arms.

At that volume, comparing three policies at one percent sensitivity, even against the more forgiving four percent baseline, costs about four months of everything ASOS sells, on the fantasy assumption that the whole order book routes into the test. Push the field to fifty and the same fantasy assumption eats the better part of five years.

No retailer commits its full order book to a single test cell. Cut the allocation to a quarter of traffic and the three-policy comparison alone outlasts a trading year. The fifty policies are not a portfolio of options; they are a queue.

Kohavi wrote the ceiling down in 2015, when candidates were still hand-built: at twenty percent of users, Bing “could only run 5 disjoint variants”.

Bing’s 250 experiments on a typical day are not the counter-example they look like. Those overlap across separate surfaces, each asking an independent question, while fifty candidate rankers for one slot compete for the same conversion events.

The strongest objection is that comparison has already been made cheap offline. Replay the logs, score fifty candidates on historical interactions, promote the best three, and the bill collapses. For that to hold, offline ranking would have to preserve online ordering — the best-scoring candidate on logged data would have to be the one that moves money. Booking.com tested the proposition across 23 comparisons of a proven model against a challenger. The correlation between offline performance gain and business value gain came out at minus 0.1, with a 90% confidence interval running from minus 0.45 to 0.27. The correlation held only where the offline metric already tracked the business metric almost exactly, which was rare in a live ranking system.

The narrowness of that finding is the point rather than a defect in it. Booking’s models were already good, aimed at one metric inside one product, the position every mature retailer occupies when it lines up fifty candidate rankers. Taobao’s team is instructive: MetaStrategy trained in a replay environment that re-runs logged requests through the live stack without exposing users, and still bought a seven-day randomised test to post a 2.11% gain in click page views. CASP, an April arXiv paper on offline policy selection validated against simulations and a reconstructed MovieLens catalogue rather than a live retail deployment, names the mechanism: swapping the candidate generator changes both a policy’s value and the data support its estimate rests on. Offline screening narrows the field; it does not settle it, and the settling is what costs.

Research is now attacking the comparison budget directly, though not from inside retail. Dynamically Allocating Evaluation Effort for Model Ranking, posted on 4 August, treats multi-model evaluation as best-arm identification with correlated arms, sampling adaptively so the budget concentrates on models still in contention. Its setting is human evaluation of NLP systems, not shoppers, and its optimality proof concerns annotation hours. The structure maps onto a retail A/B/n programme anyway, because the problem underneath is identical: costly rollouts, a fixed budget, a field of candidates that mostly lose. What does not map is the currency, since a retail rollout is paid in customers who received the losing policy.

Fifty policies is not a strategy; it is a bill nobody has read. If candidate generation keeps getting cheaper while order volume does not, the advantage moves to retailers with the best questions rather than the best models. Naming the three comparisons worth a year’s sensitivity is a merchandising judgement in a statistician’s coat, and it belongs to people who know what the business is trying to move. The forty-seven policies that never get tested are not lost opportunities. They are what three good questions cost.