Published: July 2, 2026 ｜ Source: TechCrunch, Cloudflare Official Announcement, Cloudflare Radar
While you're reading this article, more than half of all global web traffic comes from bots. Cloudflare co-founder and CEO Matthew Prince stated clearly in his latest announcement: "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge."
This inflection point was originally predicted to arrive next year. But in reality, the moment bot traffic officially surpassed human traffic has already arrived — meaning your website server today handles more than 50 bot requests for every 100 total.
According to the latest TechCrunch report, Cloudflare has formally issued a deadline notice to the AI industry: starting September 15, 2026, Cloudflare's default settings will automatically block all "mixed-purpose" crawlers — bots that simultaneously serve traditional search, AI Agent operations, and model training.
Specifically, the following will be affected:
Over the past few years, major AI companies have extensively scraped public web content via crawlers to train models. For website owners, this means their hard-built intellectual property has been freely exploited — and search engine giants like Google, playing both search and AI roles simultaneously, enjoy a "double information advantage."
Cloudflare's announcement explicitly mentions "the world's largest search engine" (clearly Google), pointing out that it leverages established crawler channels for search, making it nearly impossible for customers to truly choose between "remaining searchable" and "not being used for AI training."
Cloudflare doesn't just block; it also offers a commercialization pathway. The previously launched Pay Per Crawl is now evolving into Pay Per Use: websites are compensated when AI companies derive value from your content, not merely when pages are fetched.
Two partners have already come online:
| Partner | Business Model |
|---|---|
| Ceramic.ai | Publishers get paid when their content appears in Ceramic's AI search results |
| You.com | Publishers get paid when You.com accesses their premium content |
Cloudflare's internal data reveals a key finding: over 50% of AI crawler traffic is spent re-fetching pages that haven't changed at all. This means not only are your server resources being wasted, but your CDN bandwidth is also paying for these redundant requests.
If webmasters properly leverage Cloudflare's new tools combined with automated blocking rules, they can significantly reduce such inefficient traffic while maintaining normal search engine indexability.
Cloudflare's new default policy is a major win for website administrators — but the real challenge lies in "execution." Most small and medium enterprises lack dedicated teams to monitor bot traffic, adjust blocking rules, or set up Pay Per Use. This is where AI-powered automated operations can play a critical role: detecting anomalous crawler behavior in real time, automatically updating WAF rules, and optimizing CDN cost allocation. Our on-premise AI ops solution can help you turn the bot threat into a manageable business opportunity during this transition period.