On July 1, 2026, Cloudflare used its second Content Independence Day to replace the old binary question — block AI crawlers or don't — with three independent switches: Search, Agent, and Training. This is not a naming exercise; it is a rebalancing of who pays for CDN bandwidth and who gets cited by AI.
Starting September 15, for newly onboarded domains, sites added by existing customers, and all free-plan customers, pages that display ads will block Training and Agent crawlers by default, while Search crawlers remain allowed. Paid plans with existing sites keep their current settings — meaning small sites on free plans with ads are the ones most likely to be auto-changed without noticing.
Cloudflare reports that AI crawlers already account for 20.3% of all verified bot traffic, plus another 6.5% of AI search bots — roughly a quarter of legitimate bot load is AI-related. Training-type crawlers make up nearly 80% of AI crawler volume, yet deliver the least value back. Teams that see only "a lot of loud crawler traffic" and block all three classes save a bit of CDN cost but also kill their AI search visibility. That trade, made silently, is exactly what a proper bot policy should prevent.
In the real world, Googlebot, BingBot and the like perform both search indexing and training-data collection — amphibious crawlers. Cloudflare applies the strictest applicable rule to them: if you block Training, these hybrid crawlers get blocked too, even if they are simultaneously doing the search indexing you want. Site owners must now decide deliberately between blocking training traffic and preserving AI search citations — and often need real measurement to know the price of each choice.
Cloudflare is also pushing its Pay Per Crawl pilot toward Pay Per Use: publishers get paid when their content actually ends up in AI-generated answers, not merely when it is fetched. So far the public partners are only Ceramic.ai and You.com — an early mechanism, not yet a revenue pipeline.
After 9/15, the binary block is no longer enough. What companies need is edge-level governance that identifies crawler class, intent, and business value: let Search through, drop Training, block malicious bots — and automatically reconcile the WAF policy against the robots.txt intent, so a single 403 or 429 never silently removes your service from an AI search answer.
Blocking invalid crawlers is not about blocking everything — it's about classifying and splitting the bill. Lafa System's AIOps service continuously identifies Search, Agent, and Training traffic at the edge: Search comes through and earns citations, Training is dropped to cut CDN costs, malicious bots are blocked outright, and the WAF policy stays reconciled with your robots.txt so you never vanish from an AI search answer because of one stray 429.