Cloudflare has announced plans to block mixed-use web crawlers that index websites for search engines and serve as AI agents. This shift marks a departure from the company’s previous policy, which allowed customers to opt-out of allowing their sites to be scraped for AI chatbots. Cloudflare is now adopting a more defensive stance by default.
CEO Matthew Prince stated, “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge.” The new policy is designed to provide website owners greater control over how their content is used by AI companies while maintaining visibility and commercial opportunities.
Starting September 15, 2026, new customers and existing Cloudflare subscribers launching new websites will default to allowing search but blocking AI training and agent use for pages that host ads. Mixed-use crawlers that do not allow site owners to control AI usage will also be blocked on ad-supported pages. Free account users will transition to these new defaults unless they opt-out before the deadline.
Cloudflare is enhancing its Pay Per Crawl feature, now called Pay Per Use. This change allows site owners to earn revenue based on how their content is used in AI chatbot responses, rather than simply on crawls. Currently, the company is partnering with Ceramic.AI and You.com, suggesting efforts to engage additional AI firms.
The policy also appears to target Google indirectly. Cloudflare’s announcement indicates that the “largest search engine has access to about 2X more information than leading AI companies,” as it integrates search indexing with AI training. Googlebot is responsible for both indexing and training, yet publishers lack options to separate content use between traditional search and AI applications.
Cloudflare aims to encourage mixed-use crawlers to adopt clearer distinctions between search and AI training functions. The new measures are positioned as a strategy to prompt Google and other companies with similar practices to revise their approaches.
