> ## Content Index
> Fetch the complete content index at: https://www.theleftshift.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Cloudflare Becomes First Infrastructure Provider to Block AI Crawlers by Default
- URL: https://www.theleftshift.com/cloudflare-becomes-first-infrastructure-provider-to-block-ai-crawlers-by-default/
- Published: 2025-07-01T14:04:28.000Z
- Updated: 2025-07-01T14:04:28.000Z
- Description: Previously, AI crawlers could scrape vast amounts of online content without prior consent.
- Author: The Left Shift Bureau
- Tags: AI News, #a42

Cloudflare has announced a major shift in how content is accessed by AI companies, becoming the first internet infrastructure provider to **block AI crawlers by default** — unless explicit permission or compensation is given by content owners. 

This move marks a significant step toward a more controlled and fair digital ecosystem for publishers and creators.

Previously, AI crawlers could scrape vast amounts of online content without prior consent. Now, new Cloudflare customers will start with AI training crawlers blocked by default, shifting the model from opt-out to opt-in. Existing users can enable this feature with a single click on their dashboard.

“Website owners can now choose if they want AI crawlers to access their content, and then decide how AI companies can use it,” the company said.

In a further move to empower creators, Cloudflare has introduced **Pay Per Crawl**, a new monetisation tool that allows publishers to **set a price** for AI companies to access their content. 

Early partners include major media and tech organizations like *TIME, Reddit, The Atlantic, Condé Nast,* and *Universal Music Group*.

“Pay Per Crawl gives publishers complete control — they can block, allow, or charge AI crawlers,” Cloudflare stated. AI companies must register, view pricing, and decide whether to pay or walk away.

With this, Cloudflare aims to establish a transparent market for quality content and ensure AI models are trained with permission, accuracy, and sustainability. 

Backed by its extensive infrastructure — powering 20% of the web — Cloudflare's bot detection tools distinguish AI crawlers from other bots and enforce rules in real-time across its global edge network.