
Clothing Search Ranks at 0.87. Asked How Close Each Match Is, It Gets 51%.
A benchmark from researchers at Glance finds that off-the-shelf search models score up to 0.87 when ranking short lists of catalogue products, yet grade how closely each product matches a query correctly only about half the time. By our arithmetic, a model that could not tell a product from its near variant would beat them all on the ranking measure, so retailers should ask vendors for variant-level and per-attribute scores on their own catalogues.
Neritus Vale
Ranking scores flatter clothing search: a model can score 0.87 on the standard measure yet misjudge how closely products match about half the time. FASHION-RQ, a benchmark posted to arXiv on 13 October, reports both results for the same ten off-the-shelf text-search models. The ranking score rewards putting the right family of products near the top of a short list. Clothing search has to decide whether each item is the exact match, a near variant or merely related, and the grade tests that.
Six Glance researchers built the benchmark from a catalogue’s own structure, with no person grading relevance. Glance’s app, TechCrunch reported, dresses an AI likeness of the shopper in outfits and earns affiliate fees. The catalogue, a public Kaggle set that the authors call Myntra’s, holds 44,446 products, from shirts to perfume. A model files each product under a three-part code running from coarse to fine, of the kind generative recommenders spell out. Gemini wrote the queries “a real shopper would type” for each group. The grades follow from the code: an item sharing all three parts is exact, the first two a sibling, only the first related, none irrelevant.
Siblings are clothing’s near misses. Training pushed the second part toward product type and the third toward attributes such as colour, gender, season and price tier. In the diagram’s tree, a red T-shirt sits beside the blue one the shopper asked for. Our reading is that a sibling is usually the right garment in the wrong version.
On the ranking measure, each query brought some fifteen products, roughly half graded merely related or irrelevant, and nDCG@10 scores the top ten. The paper gives no gain function; by our arithmetic, a random shuffle of such a pool scores between 0.55 and 0.62, depending on the convention. A model that ordered the grades perfectly, except that it treated exact matches and siblings as one, would score between 0.89 and 0.95.
That model, unable to tell the blue T-shirt from the red, would beat every system in the paper on this measure.
The grading task removes that cushion, because a near miss earns nothing. Each query brought about eight products, split evenly across the four grades, so guessing is right one time in four. The best model, Qwen3-Embedding-4B, graded 51% correctly. The paper does not say how similarity scores became labels.
The sibling test isolates the hard part. Each query set one correct product against three or four of its siblings, and the score asks whether it lands in the top three of five. A random shuffle of five manages that 60% of the time. The winner at 83%, SPLADE++, is a sparse model built on BERT for general passage search. Qwen’s 4B embedding model slightly beat its 8B sibling here, though the 8B ranks first on the paper’s weighted relevance score, so size did not reliably help. The paper says it reports @1 scores here, but Table 2 shows only @3.
The attribute scores show where the misses fall. Product type and gender, which the catalogue records in structured fields, scored above 0.93 for the stronger models. Material had to be extracted from free text, and it sat near 0.52. These are match scores, not hit rates: a nearby shade earns part credit. Style barely moved across all ten models, even one never trained for search, so that column probably reflects the scoring.
The answer key needs the same reading, because relevance is whatever the code model grouped together. A cluster survived if at least four in five of its items shared a product type. Every item in a surviving cluster counts as exact for all of its queries, and the prompt asked for a separate query per main product type. A query aimed at one type can therefore count the other types as exact. The paper’s own table puts weighted colour purity at 87.9% at a codebook size of 64. The authors concede the derived structure “approximates rather than replaces human judgments”.
The human study checked the queries, not the labels. Thirty-two raters scored 20 queries from each of three sources: the benchmark, a generic LLM and real shoppers. The real ones, labelled CHAT, came from an unnamed source and had been screened partly by LLM judges voting on intent.
Benchmark queries were comparable to real ones for naturalness and beat them for intent clarity, 3.91 to 2.83 on a five-point scale. The authors take this as support for “validity for real-world evaluation”. That gap is also the distance between the test and real search, where the lowest-rated query in the study, at 1.66, read “Dark green pieces.” Screening for intent means 2.83 may flatter real search.
The strongest objection is that the 51% indicts the answer key, not the models. No person checked the clusters, so even a flawless model would disagree with some labels. A strict four-way grade punishes each disagreement in full, where a ranking measure forgives a near miss. For the thesis to fail, the ranking score would have to describe what a shopper sees in a live shop.
It does not, and the noise argument cuts both ways. A live catalogue does not hand a model fifteen candidates, half of them from other branches; it hands it every T-shirt in stock. The sibling test, also a ranking, comes closest to that setting. If the labels are too rough to grade a model’s variant calls, they are too rough to certify its 0.87.
Retailers can borrow the benchmark’s design before anyone can rerun it; the code and data arrive only “upon acceptance”. They can ask vendors for sibling and per-attribute accuracy on their own catalogue, against labels a merchandiser has checked, and for how often the right item comes first. A vendor quoting nDCG on a short list is reporting how well the model avoids the wrong shelf. Retailers can accept that number, or ask which shirt on the right shelf their shopper gets.