
Snap Lets an LLM Decide Which Ad Models Get Tested. Its 0.97 Rests on Six Data Points.
A Snap paper due at CIKM says the LLM judge that screens its catalogue-ad retrieval models "strongly correlates" with business metrics. The headline 0.97 is computed across six retrieval sources and two of them carry it; day by day the agreement is modest, and the four A/B tests behind the judge were all wins, so the paper never shows the gate stopping a model.
Neritus Vale
Snap describes using a fine-tuned LLM to screen new retrieval models for its catalogue ads before they reach a live test. The paper describing that judge says its score “strongly correlates” with business metrics, and its headline evidence is a 0.97 correlation with conversion precision. That figure rests on six data points, and two of them carry it. At a finer grain, the judge tracks conversions only modestly, and every model in its A/B evidence was a winner.
The EGR paper we covered in August described a retriever; this one, due at CIKM next month, describes the instrument that grades such models. A fine-tuned Gemini 2.5 Flash scores each product retrieved for Snap’s Dynamic Product Ads (DPA) against intent groups built from the user’s views, cart additions and purchases. Zalando reported in 2024 that its search judge reached accuracy comparable to human annotators at up to 1,000 times lower cost than human labour. Snap also reports a human-audited set (F1 0.869), but rests its case on online business metrics, “on the launch-decision critical path”.
Take away two outlying sources and the correlation all but disappears. Table 1 compares six: a retargeting source drawn from the user’s own engagement log, a collaborative “similar user” model, and popularity, content-based, engagement-based and random retrieval. The first two sit far above the rest on both conversion precision and the judge’s score, so they pin the fitted line. By our arithmetic, the same correlation across the other four is 0.13. Four points prove little either way, but they show the 0.97 depends on the two.
The retargeting source wins on both conversion precision and the judge’s score almost by construction. Conversion precision counts a hit only when a retrieved product is “identical to the converted item” within a day of the impression; the paper calls this source’s precision “high by construction”. The judge scores against intents built from engagement history, so presumably a product the user has just viewed is its easiest “Exact” call. Agreement here is agreement that retargeting works, which no performance marketer needed a language model to learn.
Day-level agreement is weaker. Measured day by day over a month, with date-and-source pairs pooled from two retrieval sources, the relevance score correlates with conversion precision at 0.35 under production settings. The analysis uses labels from the larger Gemini 2.5 Pro teacher model, not the deployed Flash. The bootstrap interval runs from 0.04 to 0.63.
Cutting each user’s history to the last day barely moves the correlations. The paper says that cut “halves intent-set fidelity”, yet the shift is no more than 0.06. The authors say the business signal “is concentrated in a small, recent slice of engagement”, which is the premise of retargeting.
Online, the paper claims only “directional agreement” across four A/B tests, all of them wins. Each new content or engagement model raised the judge’s score, then beat its predecessor on click-through and conversion rates, significantly. A gate earns its keep by stopping a model that would lose, and the paper shows none. A score that rose for every new model would have passed the same test.
Across the two model families, the sizes of those wins do not line up with the judge’s MicroRel score. The first engagement model scored 0.068 against 0.138 for the first content model, half the offline gain, yet delivered a 0.350% click-through gain against 0.149%. On MicroRel, the ordering within each family matches the live results. The paper acknowledges “relatively weak correlations” for engagement-based retrieval, so the judge is weakest on the family tuned hardest towards conversions.
A July Snap paper found a different LLM judge ranking systems opposite to a purchase metric. SMART, which shares nine of these twelve authors (one appears under a different first name), used Gemini 3 Flash to score “prospecting”. Set beside a category-level purchase recall measure, it produced “opposite rankings on the same systems”. SMART’s authors built the two metrics for different objectives and read the inversion as confirming it. Read next to this paper, it suggests an LLM relevance score agrees with purchase metrics when retargeting separates the systems, and parts company with them when discovery does.
A Snap agent wrote the judge’s rules and missed demographics. It kept only rules that significantly lifted accuracy, and surfaced seven of the twelve reasons in Snap’s editorial guideline, size and colour among them, but not gender or age. Human editors promoted demographic mismatch from a “Substitute-leaning signal” to a hard “Irrelevant” override, a rule the loop had “repeatedly deprioritized because demographic-mismatch pairs are statistically rare in the calibration set”. In apparel, that rare pair is, say, a men’s jacket shown to a womenswear shopper.
The gate’s validation shares that blind spot: six sources and four winning tests never show it meeting a model that looks better offline and loses online.
The strongest objection is that the judge only screens, and the A/B test still decides. If every model it passes faces a live test, a false pass costs a test slot rather than revenue. Its best day-level correlation, 0.50 with conversion rate, is worth having before paying for an A/B test, since conventional offline metrics suffer from “sparse, biased feedback”. The objection holds only if Snap also tests some models the judge marks down, and the paper reports no such tests.
The abstract promises “faster and more reliable DPA retrieval iteration before A/B testing”, and the conclusion calls the metric “the offline gate for DPA model evaluation before online deployment”. It reports no measurement of the models the gate wrongly rejects.
Catalogue advertisers presumably bear the cost, because their products reach shoppers through whichever model clears the gate. If Snap keeps tuning the judge to track conversions, the gate would reward retrieval that resembles retargeting over models that show shoppers something new. Snap could settle this by publishing the judge’s verdicts on models that lost live tests. Retailers can take “strongly correlates” on trust, or ask Snap to show the gate stopping a loser once.