Cloudflare Becomes First Infrastructure Provider to Block AI Crawlers by Default

Cloudflare has announced a major shift in how content is accessed by AI companies, becoming the first internet infrastructure provider to block AI crawlers by default — unless explicit permission or compensation is given by content owners.

This move marks a significant step toward a more controlled and fair digital ecosystem for publishers and creators.

Previously, AI crawlers could scrape vast amounts of online content without prior consent. Now, new Cloudflare customers will start with AI training crawlers blocked by default, shifting the model from opt-out to opt-in. Existing users can enable this feature with a single click on their dashboard.

“Website owners can now choose if they want AI crawlers to access their content, and then decide how AI companies can use it,” the company said.

In a further move to empower creators, Cloudflare has introduced Pay Per Crawl, a new monetisation tool that allows publishers to set a price for AI companies to access their content.

Early partners include major media and tech organizations like TIME, Reddit, The Atlantic, Condé Nast, and Universal Music Group.

“Pay Per Crawl gives publishers complete control — they can block, allow, or charge AI crawlers,” Cloudflare stated. AI companies must register, view pricing, and decide whether to pay or walk away.

With this, Cloudflare aims to establish a transparent market for quality content and ensure AI models are trained with permission, accuracy, and sustainability.

Backed by its extensive infrastructure — powering 20% of the web — Cloudflare's bot detection tools distinguish AI crawlers from other bots and enforce rules in real-time across its global edge network.