In 1997, the IETF defined an HTTP status code that has remained largely dormant ever since: 402 Payment Required. It was designed as a placeholder — technically available but virtually unused. Nearly thirty years later, this sleeping status code has become one of the most discussed topics in web infrastructure.
The reason is straightforward: AI crawlers are consuming website operators' resources and budgets at an unprecedented scale, and HTTP 402 offers the first standardized way for machines to negotiate "pay-per-access" agreements. Notably, the companies leading this change are not startups — they are Cloudflare and AWS, two of the largest cloud infrastructure providers in the world.
In July 2025, Cloudflare launched Pay Per Crawl. Website owners can set a per-request price for their domains, and when an AI crawler requests content, it must include payment intent in its headers or receive an HTTP 402 response and be blocked.
On June 15, 2026, AWS followed suit by integrating the same mechanism into WAF Bot Control. AWS's implementation is even more granular: pricing can be set per content path and per bot type, without requiring any changes to website code. Payments settle in stablecoin through Coinbase infrastructure.
The common thread between both systems is clear: the billing mechanism lives at the CDN/WAF edge layer — the same layer that already decides "who gets in." This means access control and monetization have formally merged, ending the era of default-allow for all traffic.
On the surface, HTTP 402 looks like a way to "rent out your content" to AI companies. But looking deeper, the real issue is not revenue — it's visibility.
For thirty years, there was an implicit agreement between search engine crawlers and websites: let me crawl your content, and I'll send visitors back. Every time Google Bot indexed a page, that page could appear in search results, driving real traffic. It was a mutually beneficial cycle.
The AI era broke this agreement. AI crawlers do not build "clickable indexes" — they extract content for model training, taking your data away but rarely sending readers back. Cloudflare's analysis shows that roughly 80% of AI crawler activity is for model training, with search-purpose requests that actually generate referral traffic accounting for only a tiny fraction.
The essence of HTTP 402, then, is: letting website owners decide "which bots earn free access to my content." Choosing which crawlers to charge or allow is essentially choosing which AI responses will reference and promote your brand.
"Should I charge AI crawlers?" There is no single answer. The right strategy depends on your content type and distribution model:
Both Cloudflare's and AWS's HTTP 402 implementations are built on the same underlying protocol — x402, an open standard led by Coinbase. It enables machine-to-machine payment via stablecoin, requiring no account or KYC from the paying agent. This means the toll booth can extend beyond known AI crawlers to anonymous bots as well.
Third-party solutions like TollBit, Akamai, and Skyfire are also entering this market. The shared trend across all of them: access control and billing happen in the same edge rule engine. You no longer need separate WAF firewall and billing systems — they have converged.
Whether you decide to charge AI crawlers or not, HTTP 402 represents one reality: your WAF/CDN edge is now the only frontline for controlling bot traffic. Every operations team needs to do three things:
First, see who's crawling you. Use bot analytics to identify which crawlers are search engines (welcome them), which are AI agents (potentially valuable), and which are pure training bots (take without giving back). What you cannot see, you cannot manage.
Second, establish tiered policies. Set different access rules for different bot types — full access for legitimate search engines, rate limiting for AI agents, blocking or charging for training crawlers.
Third, automate enforcement. Manual monitoring and manual blocking are no longer sufficient. You need an automated defense system that can identify, classify, and act on traffic before it ever reaches your servers.