
Google Shows Shops How to Let Its Agent Skip the CAPTCHA. Saying No Takes a Firewall Rule
Google's new guide shows websites that want Google-Agent how to verify it and spare it bot challenges all the way to the basket. The agent generally ignores robots.txt, so a shop that wants it out must refuse it in the firewall, and whether to admit it is each shop's decision.
Sir John Crabstone
Google-Agent generally ignores robots.txt, so a shop that wants it out has to refuse it in the firewall. That is the reading in Hello Partner’s report on the guide Google’s changelog dates to 17 November. The guide spells out how to verify the agent; whether to admit it is each shop’s decision.
Google lists Google-Agent among its user-triggered fetchers. The name covers agents running on its servers “to navigate the web and perform actions upon user request”. Such fetchers “generally ignore robots.txt rules”, Google says, because a person asked for the fetch; on that logic, a person’s request usually outranks the shop’s notice.
The guide gives three tests of the agent’s identity: a User-Agent header containing “Google-Agent”, an address on Google’s published list and, where provided, a signature on the request. Google says it is experimenting with signing. Then comes the instruction that matters: “For verified requests, skip bot challenges.”
If the agent is still blocked, the troubleshooting list says its allow rule should rank above data-centre blocks, managed bot rules and “category-level AI agent block policies”. The agent should also skip in-page CAPTCHAs and be covered on search, filters, add-to-cart and the APIs beneath. Because “concurrent requests from multiple users share the same egress pool”, per-IP rate limits should make room for Google’s addresses.
Some of the web’s biggest sites are refusing the agent in a file it generally ignores. As of 20 November, Known Agents, a bot directory that also sells bot blocking, finds 7% of the top 1,000 websites it scans daily blocking Google-Agent in robots.txt; the sample is not restricted to shops. It advises most sites not to block the agent, and notes that “a disallow rule only communicates your preference”.
Hello Partner has the mechanics right. “The control point is the firewall, not robots.txt,” it wrote. For an agent that gives its name, the firewall is where a shop makes its choice.
Another test used a different agent. In i22’s test of 50 of Germany’s biggest online shops, which we reported on 20 November, the agent was Claude in Chrome in a normal browser profile, not Google-Agent. i22, a consultancy that sells agent-readiness services, found it rarely stopped as a bot.
Bot protection intervened in 5 of 147 runs, i22’s study says. The study adds that nobody outside can tell whether shops admit such agents on purpose or their bot protection misses them. The two figures measure different things: robots.txt rules on the top 1,000 sites, and bot-protection stops in 50 German shops.
Google-Agent gives its name in every request and arrives from addresses Google publishes. The name alone proves little, since Known Agents notes that any bot can claim it, so the address list does the checking. The same two signals serve a shop that wants to refuse the agent.
Google’s fetcher list no longer names any product that sends the agent. A 9 July archived copy of its entry gave one example, Project Mariner; the current page gives none. A shop that admits the agent is not told which Google service it is dealing with.
Google’s changelog says the guide exists to “help publishers that want to enable Google-Agent”. Of how to refuse the agent, the guide says nothing.
The guide’s troubleshooting list runs to six checks; the seventh, whether to want the agent at all, Google leaves to the shop.