Research & TrendsDeep Dive (Vale)
A conveyor belt carries filmstrips of sneakers, serum bottles and wristwatches from a machine lettered ANOKI to one small desk, where a reviewer holding a magnifying glass and a REJECT stamp sits beside a brass robot sticking red flags on the strips, while a nautilus watches from a tall pile of rejected strips.

Anoki's Review Rules Would Reject 117 of Its 120 AI Product Ads

With AI judges' findings on screen, Anoki's reviewers marked the product wrong in most scenes of the company's generated video ads, and its review rules would reject 117 of 120. Reviewers without the findings marked less than half as much, which puts the limit on an ad for every SKU at the review screen, a cost that 'ad-ready' sales pitches leave out.

Neritus Vale

One ad in Anoki’s study sells a skincare serum, and reviewers marked the serum wrong in every generated scene, from a misdrawn logo to garbled ingredient text. Its narration turns users’ answers in a company survey into an observed result, “Ninety-one percent saw faded marks”, and a script judge flagged the change. Anoki’s numbers suggest that the limit on a pipeline built to “make an ad for every catalog product” is the checking.

The paper, “Multimodal Judges Flag, Humans Decide”, was posted to arXiv on 9 November by researchers at Anoki, a connected-TV ad-tech firm. Its pipeline turns a product’s listing and catalogue photos into a short ad with script, narration, scenes and captions. The test was one review round of 120 ads “in one market, with English narration”. Anoki staff reviewed them with findings from AI judges that run on closed models.

The ads broke on the product. By majority vote, script, caption and sound problems each turned up on under 10% of items. The words and sound came out mostly right; the thing for sale did not. A majority marked the product wrong on 60% of scenes. That is the error a shopper can check by opening the parcel.

Under Anoki’s rules, almost no ad survives those marks. A product marked wrong in any scene rejects the ad. By the majority’s marks, which on 40 of the 120 ads were a single reviewer’s, that rule alone would reject 117 of the 120 before any edit. The authors call such thresholds “policy choices”, and a person can overrule the verdict.

Neither sales pitch mentions checking the product against its listing. Amazon’s Video Generator launch note, dated 19 November 2025 and marked “Open Beta”, promises “professional video ads within minutes” from a product detail page and calls them “ad-ready”. Anoki’s CreativeAI “delivers ready-to-run video ads” and offers edits “before it’s made or after”. The paper tests neither tool, does not name the video models that made its ads and never says whether its pipeline is CreativeAI. What it tests is the word “ready”.

Amazon’s note offers advertisers control over headlines, fonts, colours and logo placement, and lists none over the product on screen.

Without the judges, product marks fall to less than half. Two reviewers went through a subset of the ads with every judge hidden and marked the product wrong on 26% of scenes on average. The round’s reviewers, who included one of the two, had seen those scenes with findings on screen. Their majority had marked 58%.

Reviewer R2 saw the same ads both ways, findings first, which holds the person fixed. With the findings, R2 marked the product wrong on 57% of scenes; without them, 19%. Memory of the findings should, if anything, have raised R2’s unaided marks and narrowed that gap.

Two reviewers are a thin base. Unaided, they marked 33% and 19% of scenes, and the authors say a crossover with more reviewers is needed to separate reviewers who find defects with the judges’ help from reviewers who only follow the flags.

R2’s unaided marks also faded over the session, in an exploratory analysis. They started at 25% of scenes and ended at 12%, while reviews sped up. The later ads were no cleaner: with findings, R2’s marks on them held steady. The other unaided reviewer also sped up, yet their marks held steady.

Cheap automatic scores barely see the problem. A product judge that watches each clip beside the catalogue photos separated defective from clean scenes with an AUC of 0.84 on blind labels. The authors call its lead “optimistic”, since they designed it on those labels. Photo-similarity and video-quality scores stayed near chance. One of them, VideoScore2, gave 107 of the 120 clips its top visual-quality score, defects included; but it and DOVER never see the catalogue photos, so neither can tell a wrong product from the right one.

Anoki tunes the judge for recall, so a person checks every flag. On blind labels it caught 26 of 29 defective scenes. Under half its flags matched a labelled defect, a price the authors accept because “a false flag costs one click to reject”. Target’s MOSS monitor made the opposite trade, calibrating its judge partly to what “the human triage queue can sustain”.

The strongest objection is that the judges manufacture the marks. Reviewers accepted 75% of the product judge’s findings, well above its hit rate on blind labels, though the authors think the round held more defects. They concede they “cannot tell how many added marks are right or how much acceptance is over-trust”. If most extra marks are deference, unaided reviewers were not undercounting, and the fix is a better judge, not more review.

That reading moves the error but leaves the cost where it was. A judge whose flags are often wrong needs a person to reject each one, and the portal blocks submission until every finding has a thumb. Where the flags are right, unaided review misses them; even the blind labels that score the judge came from annotators who “miss subtle defects”. A precision-first setting swaps one cost for the other, catching only 15 of 29 defective scenes.

The round’s findings name no garments, though its 20 products include a sports shoe. One ad for the shoe opens with a giant translucent shoe floating through the model’s body. Yet the paper’s first example of an invented detail is “an extra pocket on a pair of pants”, and its supplement shows kurta and trousers ads from outside the round. In the round, one smartwatch’s strap buckle is bright silver for a second; the real one is dark blue. If generated clothes drift like that, every colourway will need its own review.

Editing is where the paper is thinnest. Repair, a higher-resolution re-render, fully fixed garbled product text in 18 of 26 rated scenes, as one internal reviewer judged. Any other product fault needs a borrowed, new or edited scene, and the edited ad is “reviewed again like a new ad”. The share of ads delivered after edits, the number that would price an ad per SKU, is future work.

If generation stays nearly free and review stays human, a catalogue’s video budget moves from the studio to the review queue. A retailer can staff that queue: at the paper’s assumed $20 an hour, a review costs about $1.40, and 78 of the round’s 120 ads got three. Or it can run the ads unchecked, and a shopper may open the parcel to find a pocket that was never there; the paper reports no such case.

Related Coverage