
Shopify Deployed Generative Search. The Lift It Published Is the Home Feed's.
Shopify's SID-LLM report says A/B tests supported putting its generative retriever into search, but the only online lift it prints is the Shop app's home feed, with no traffic share or test length. The model reads products only as three-token codes, and the report does not say whether a garment's photo shapes its code.
Neritus Vale
Shopify has put a generative retrieval model into its product search and published no online figure for it. Its technical report, posted on 5 November, says A/B tests backed the deployment; the only online lift it prints comes from the Shop app’s home feed. For clothing, the missing detail matters more: the model “can only ingest text and SIDs”, so it reads a dress as a three-token code.
The model, SID-LLM, takes a query or a shopper’s history and writes product codes, which an index maps to eligible products. Each code, or semantic ID, is three tokens quantised from one product embedding, using codebooks fitted on the catalogue’s highest-GMV products. Its backbone, LFM2, comes from Liquid AI, as does one co-author, Anthony Maltsev.
Shopify’s search runs on more than one surface. The company “serves product search through the Catalog API, merchant storefronts and the Shop app”, and the report describes the Catalog API as “designed for conversational commerce and AI agents”. An appendix analysis uses August logs of “standard and agent search sessions”, but it tests how well the codes cluster, not how the model performs in deployment. Which of the three surfaces the A/B tests covered, the report leaves open.
The online evidence for search is a list of metrics and a verdict. The authors judged “several successive versions” on purchases, high-quality clicks and clicks, overall and for head, torso and tail queries. The verdict: “The results of these evaluations supported deploying SID-LLM in production as part of Shopify’s search stack.” No figure follows.
The report names three search surfaces, one of them the Catalog API built for agents, and prints an A/B number for none of them.
The abstract reads more generously, claiming for both stacks “significant gains in both offline evaluations and online A/B tests”. In the body, the only online figures behind that claim are the home feed’s.
Shopify has announced search models before, leading with speed. Liquid AI’s November 2025 partnership release called the first deployment a “text model for search” answering in under 20 milliseconds. The co-developed recommender was credited with “higher conversion rates from recommendations”. Digital Commerce 360 reported the search model “now live” on merchant storefronts. The report does not say whether that was an early SID-LLM.
Offline, the search evidence rests on a judge. In Table 17, removing SID-LLM from the full search stack lowers OneDCG@10 from 0.3269 to 0.3057. SID-LLM alone scores “close to” the stack without it, in the authors’ words: below it overall, above it on personalised queries. OneDCG mixes an LLM judge’s grade with a static quality score built from ratings, popularity, brand prevalence and shop sales, so the offline gain is judge-scored. Purchase recall, which the authors call “the more independent of the two metrics”, is missing from that comparison.
The online numbers come from recommendation, where SID-LLM is one candidate generator among several. In the home feed, adding it raised attributed orders by 7.95% on what the report calls “the impacted surface”. The number of converters and the conversion rate rose by more, and the authors call every lift significant after Dunnett correction.
Table 19 omits what would size those figures. It gives no traffic share, no test length and no definition of the impacted surface. The correction it cites, Dunnett’s, was built for “comparing several treatments with a control”; the table prints one treatment.
Shopify’s February engineering blog offers a loose yardstick: a recommender that began as an HSTU build lifted Shop orders by 0.94% in an August model version. That measures Shop orders for a different model, while the home-feed figure counts attributed orders on a surface the report never defines. A lift that much larger could mean a better model or a narrower base.
Offline, Table 18 supports the better-model reading: SID-LLM reaches a Recall@50 of 24.61 against HSTU’s 17.21. The online lift could still sit on a narrow base. Without the base, no reader can tell.
Xiaohongshu’s RedMall report, dated 26 October, feeds images into its codes. Its codes come from a text embedding and a visual embedding, fused by a learned gate. The public benchmarks are Amazon ratings for beauty, toys and food, plus MovieLens films. Neither paper reports a result for clothing.
RedMall prints the terms Table 19 lacks. Its recommendation test gave the new model a tenth of traffic against an equal control for seven days. Every metric in its results table is significant, and GMV rose 1.08%. Shopify’s larger figure has neither a traffic share nor a duration.
A dress reaches SID-LLM only as its code. The model learns what codes mean from clicks and from text. Its training pairs each code with one attribute at a time, such as title, category, brand, price or shop name. The embedding behind each code comes from “an in-house model trained for product search”, and nothing in the report says whether the photograph feeds it.
Shopify’s own documentation says what a photo adds. Its agent catalog accepts an image for “visual context, such as style, shape, or pattern”, a fair list of what separates one dress from another. The report never mentions that route. In November 2025, Liquid said the partners were “evaluating multimodal models” for customer profiles, agents and product classification; the authors now list “multi-modal understanding of products” as future work.
The strongest objection is that the deployment decision is itself evidence. Shopify tested successive versions across head, torso and tail queries, and keyword and embedding retrievers still run beside SID-LLM, so the model had several chances to fail. For the thesis to fail, the aggregate gain would have to hold for every kind of listing.
Aggregates hide losers. The report splits search tests by query frequency, not category or shop, so a net rise in purchases can coexist with losses for new garments from small shops. When products share a code, SID-LLM passes on only the one with the highest static quality score. By the report’s Table 31, 77.2% of products share a code; 22.8% sit alone. Only the largest clusters the authors inspected were identical or near-identical listings. In October we traced popularity’s share of another Shopify paper’s generative edge; here it breaks ties.
The fix is a cut Shopify already makes. It measured its search tests separately for head, torso and tail queries. Published, and cut by category too, those results would show apparel merchants what the model’s reading of their listings is worth. Until then, the model reads their listings, and nobody outside Shopify has seen what that does to their sales.