
Tencent's Agent-Evolved Recommender Climbed Four Rungs of Evidence, and Only the Last Had Live Users
In Tencent's TGR-EvoDAG paper, LLM agents evolved a generative recommender on one small Amazon dataset, and the result, retrained for a media platform, lifted click-through 0.40% in a 14-day live test. The bigger numbers come from replayed reviews and from simulated users run on the same model that wrote the critiques the agents read while evolving, which is the evidence a retailer without Tencent's traffic would be left with.
Neritus Vale
Researchers at Tencent let LLM agents rewrite a recommender’s training code, and the change survived a live test. Their paper, TGR-EvoDAG, submitted to arXiv on 23 October, reports the gain on four rungs of evidence ending in that test, and the number falls on every rung. A retailer that cannot run such a test would be judging agent-written changes on the first two rungs, the cheapest to produce.
The agents worked on a generative recommender, trained in three stages. A tokenizer turns each item into a short code, a model learns to write the code of a user’s next item, and an alignment stage tunes its choices. Because tokenizer edits force a full retrain, the system budgets GPU time by stage and keeps rival branches alive. Seven agents share the work, one of them a code reviewer.
The first rung is a replay of one small Amazon dataset. The abstract speaks of “public Amazon datasets”, but the appendix puts every evolution experiment on the Office Products category, with 4,905 reviewers. No apparel category was tested.
On that data the evolved model beat APAO, the strongest human-designed method tested, by 37.4% on NDCG@10, a score for how high the right item sits in the top ten. But the agents started from APAO’s released code, stacked changes on it and merged the APAO branch back in at the end. The comparison measures how far agents can push published human work.
The paper prices that search in GPU-hours, 24 for each framework compared. The agents’ language-model tokens are logged outside that budget and not reported, and no figure is given for the human research behind APAO.
The second rung swaps logged reviewers for simulated ones. Agent4Rec builds LLM agents from real users’ profiles and has them page through recommendations, rate them and decide when to leave. A panel of these agents scored each framework’s final model, and their satisfaction rose 19.7% over APAO. The authors say this confirms “genuinely better recommendations”.
That check shares an engine with the search it grades. The paper says Agent4Rec scores were never used to select models and calls them a “genuinely held-out behavioural measurement”. Even so, one panel of simulated users scored the final models and another wrote the critiques the agents read while evolving; both ran on DeepSeek-v4-flash. The panels are kept apart, but they still speak for one model, whose preferences steered the search before they scored its result.
Simulators of this kind carry their own warnings. Agent4Rec’s authors reported “deviation” as well as alignment between their agents and users’ preferences. When simulators replayed one live A/B test in UserArena, their readings ran from almost nothing to an overstatement, as we reported on 23 October.
The third rung moves to Tencent, though not to a shop. The authors retrained the evolved architecture, and APAO, from scratch on logs from what the paper calls “Tencent’s large-scale commercial media platform”. It has more than 10 million daily active users, and a positive interaction there is a click; the paper reports no purchase measure. The agents never worked on this data.
On those logs the abstract quotes a 12.16% gain in Hit@1, how often the clicked item was ranked first. That is the widest gap on the paper’s chart of cut-offs. At Hit@10, a cut-off the Amazon tests also used, the gain is 1.52%. A retrieval channel hands candidates to a ranker, so presumably its first slot matters less than whether the item makes its list.
The fourth rung is the only one that measures people reacting to the new system. Run for 14 days as one extra retrieval channel, the evolved model raised overall click-through by 0.40%, with p=0.019 after variance reduction and a correction for multiple comparisons. The paper withholds absolute click-through values, so the 0.40% cannot be turned into clicks or revenue. Tencent’s earlier TGR paper reported reading and watch time alongside clicks; this one reports no measure of time spent.
That test drew on 5% of live traffic, more than a million users. A retailer with a fraction of that traffic would need a far larger effect before a test could tell it from noise. The paper also offers no exchange rate between rungs, since each measures something different.
The handover from evolved code to production is where the paper says least. Tencent’s setup differs from the Amazon one, with user histories of up to 500 interactions and item text that includes machine-written captions of images. The paper does not say what changed in the port, or whether an engineer read the evolved code before it went live.
The only code reviewer it describes is another agent.
The strongest objection is that none of this is peculiar to agents. Tencent’s BARGE model, from the same TGR paper, improved offline Hit@5 by double digits. After full rollout, it lifted click-through by 0.60%, so offline scores flattered a human-built model too. The thesis fails if the cheap rungs at least rank candidates in the order a live test would, so a retailer could pick winners without knowing their size.
One live test cannot show that ordering, and the paper ran exactly one. The agent system Google researchers run at YouTube assumes the ordering can fail: its offline agent steers away from “dead-end trajectories that appear promising in offline proxies but have historically failed to improve online metrics”. The engineers’ final step is reviewing “experiment metrics collected by the Online Agent”. Both depend on live tests, which a retailer starting out lacks.
TGR-EvoDAG runs inside Uppimo, an internal Tencent platform that the paper says will open for public trial. If it does, retailers could use the half of this process that agents made cheap, the offline search for better code. The other half, a live test big enough to see a 0.40% move, does not come with the software. A retailer that adopts the cheap half without building the live test has chosen to ship on the paper’s two cheapest kinds of evidence.