For years, website owners had only one crude tool to deal with AI crawlers: block all bots or let them all through. This binary approach changed meaningfully in July 2026.
On July 1st, Cloudflare rolled out three independent AI traffic switches to every customer (including the free tier): Search (search engine crawlers), Agent (real-time AI proxies), and Training (model-training crawlers). Starting September 15th, defaults will flip — any page serving ads will automatically block Training and Agent-type bots.
In the old world, robots.txt was the only statement of intent, but most AI crawlers simply ignored it. Googlebot simultaneously scrapes content for search indexing and Gemini model training; Anthropic's ClaudeBot crawls over 11,000 pages before generating a single referral click (Cloudflare Radar mid-2026 data); OpenAI's ratio is also sobering, around 857:1.
By contrast, traditional Googlebot generates one click for every five pages crawled. The message is clear: you are running substantial server resources for free to serve AI training companies, while getting almost no real visitors in return. And this invalid traffic is directly eating into your CDN bandwidth and billing quotas.
Cloudflare's three-way split finally lets web owners issue separate instructions to "valuable search engines" and "take-without-giving-back training crawlers," instead of being forced to choose between "all open" or "all closed."
The most debated part of this policy is not about blocking Training Bots, but that Googlebot is classified as a "multi-purpose crawler" — because Google uses the same crawler infrastructure for both search indexing and Gemini model training. So if you block Training traffic, Googlebot on ad pages gets blocked too.
The discussion on Hacker News was intense. Many developers argued this is exactly what Google deserves: "When OpenAI has already split GPTBot, OAI-SearchBot, and ChatGPT-User into three distinct identities, Google still uses one crawler for both search and training — the problem is obvious."
Cloudflare does offer an opt-out — you can whitelist "Training+Search" hybrid crawlers in Security settings. But this choice itself reflects a deeper industry conflict: web owners forced to keep supplying free training data just to retain Google search traffic.
The answer to "how should I set these three switches?" depends on your revenue model and traffic sources:
IETF has recently pushed "Content Signal" extensions to robots.txt (search=yes,ai-train=no,use=reference), but these declarative formats only work for companies that voluntarily comply. Cloudflare's dashboard settings are what actually enforce firewall rules at the edge.
In practice, the most effective approach is running both in parallel: robots.txt expresses your intent so compliant crawlers auto-obey; Cloudflare's panel is the bottom-line defense, blocking all non-compliant traffic. This maps directly to WAF operations logic — declarative policy plus mandatory enforcement layer.
No matter what type of site you run, Cloudflare's overhaul points to the same conclusion: bot traffic management is no longer about "whether to do it" but "how to do it without hurting yourself".
First, you must know who is crawling what. If your log analysis tool still only distinguishes humans from bots, you have zero visibility into how many gigabytes ClaudeBot consumed today, which agents are accessing legitimately, and which ones are abusing. You can't manage what you can't see — this is the prerequisite of all automated defense.
Second, you need fast decision-making at the edge. By the time a crawler request reaches your backend server, it's too late — traffic costs are calculated per request and bandwidth, not by "whether this bot has value." Your defense must classify and block at the CDN/WAF edge before data hits your servers.
Third, set-and-forget doesn't work. Crawler behavior changes (Googlebot might split into two separate UAs tomorrow), new bots will emerge, and your site's purpose may evolve. Good bot management requires periodic review and adjustment, not a one-time configuration.