#060
2026-11-10

AI Crawlers Take Everything and Send Nothing Back: Who Pays for the 1,700% Traffic Growth?

đź“° Want to read more news?
Browse the full news list

In 2026, the internet gained a second audience — and more than half of all traffic is no longer generated by humans. In its end-of-September report, Cloudflare said the average load it carries has climbed from 63 million HTTP requests per second at the end of 2024 to 115 million, and the largest share of that growth does not come from search engines but from AI systems: daily requests from AI agents grew more than 1,700% in a single year, and user-triggered fetching grew over 15x in 2025. For site operators, every one of those requests is real — real connections, real bandwidth, real invoices.

First, split the audience into three kinds. Training crawlers (GPTBot, ClaudeBot, Applebot-Extended and the like) fetch content to feed into model learning; AI search crawlers (OAI-SearchBot, PerplexityBot) fetch pages on the fly to answer user questions; user-agent fetchers (ChatGPT-User) go grab a page on behalf of someone typing in a chat box. HUMAN Security's April statistics put ChatGPT-User at 31.8% of AI-driven traffic, OAI-SearchBot at 19.5%, and GPTBot at 17.6%. Cloudflare's numbers are more direct: in June 2026, 52% of the crawler requests it received were for AI training — a share that was only 22% in spring 2025.

The problem is not “being crawled,” it is “crawled, and given nothing back in return”. Search engines run on a crawl-for-traffic exchange: Googlebot crawls your pages, ranks them, and brings people. AI crawlers have no such exchange — Anthropic's fetch-to-referral ratio peaked at 500,000 fetches for every one referred visitor in 2025, versus 3:1 to 30:1 for Google. The owner of a Linux and security guide site, published for over a decade, compared origin access logs line by line over the past few weeks: in six and a half days, AI crawlers fetched 16,194 pages, while Googlebot and Bingbot together fetched 10,433; and on one night, after the AI answer bots reached the origin and fetched 57 pages, the number of visitors they sent back was zero — not low, zero. The machines read, left, and the bill stays with the person who wrote the guide.

Site operators are pushing back, but politely. A robots.txt analysis from early August covering the internet's top 2,000 domains found that about 19% of domains now block at least one mainstream AI crawler, and Amazon blocks all five of the major ones — GPTBot, ClaudeBot, Google-Extended, PerplexityBot and Bytespider. Among Cloudflare's sites, fewer than 1% block search crawlers, while 17% block training crawlers — the prevailing stance is “let AI search bring traffic, but don't let AI training eat for free.” Yet robots.txt is, by nature, a gentleman's agreement: Fastly's January data shows that 97% of AI fetch requests on its network came from ChatGPT-User and OAI-SearchBot, and 57% of them targeted content that was not publicly available or did not exist in the first place — machines that don't even read a polite refusal treat your robots.txt block as a mere suggestion.

For operations teams, this means bot governance cannot stop at robots.txt or a CDN console toggle — it is ongoing security and cost engineering. First, identification must be trustworthy: blocking on User-Agent strings is the same as blocking air — major crawlers' UAs have long been spoofed at scale — and what actually works is CDN/WAF-level verified bot signatures and behavioral signals. Second, classification must be fine-grained: training, search and user-agent fetches differ completely in risk and value; a blanket block hurts the AI search exposure you want, a blanket allow blows up the bill. Third, cost must be visible: only by carving AI traffic out of total bandwidth and attributing it separately can you see how much of the monthly CDN spend is actually paying for machines. And one fact people often miss: Vercel's measurements show major AI crawlers barely execute JavaScript — client-side content is invisible to them in the first place. That is both a content-protection side effect and an explanation of why “57% fetch non-public content” is even possible.

đź’ˇ LAFA Perspective

What grows 1,700% a year is AI traffic — and what should be blocked is not “AI,” it is “free-riding.” LAFA's 24/7 AI operations handles bot identification and routing at the WAF layer: verified search crawlers pass and earn exposure, training crawlers and spoofers are cut off outright, and the CDN cost of paying for machines every month is carved out of the bill and made visible.