Cloudflare Now Lets Websites Block AI Training without Losing Search Visibility

Cloudflare has introduced a new set of controls that allow website owners to prevent their content from being used for AI training without removing it from traditional search results.

The company announced an “Accountable” designation for AI crawlers alongside a new “Disallow AI Training” setting.

The controls are designed to separate a website’s preferences for search crawling, AI training and AI agents, addressing a growing conflict between visibility on the web and the use of online content to develop AI systems.

Cloudflare Unveils AI Defense That Makes Automated Cyberattacks More Expensive

The company said the approach is designed to prevent attackers from studying a fixed defense and repeatedly exploiting it.

The Left ShiftThe Left Shift Bureau

Cloudflare said mixed-use crawlers, which collect content for both search indexing and AI training, now account for 36.6% of verified crawler traffic across its network. While fewer than 1% of website owners block search crawlers, 17% restrict AI training, according to the company.

Previously, websites that wanted to prevent AI training could have to block crawlers that also supported search, potentially affecting their visibility in search results.

“This is how we make the Internet better: preserving the openness that makes search valuable while giving the people and businesses behind the web meaningful control over how their work is used. We look forward to continued collaborative engagement with companies like Apple, Google, and Microsoft as we work together to build a healthy ecosystem,” said Matthew Prince, Cloudflare Co-Founder and CEO.

Cloudflare said Apple, Google and Microsoft have now been designated as Accountable after demonstrating that they meet its criteria or providing timelines for compliance.

Cloudflare Open-Sources Cloudflare OS to Bring Secure AI Workspaces to Every Employee

A managed version will be available through Cloudflare’s dashboard in the coming months, while implementation partners including Presidio and Happy Cog will help enterprises customise deployments.

The Left ShiftThe Left Shift Bureau

The requirements include providing an opt-out mechanism for AI training, allowing websites to opt out of AI-generated search summaries, offering URL-level visibility into content usage, and confirming that opting out of AI training will not affect traditional search rankings.

Earlier this year, Cloudflare revealed that automated bots now generate more web traffic than humans, marking what could be the first time in the internet's history that machines outnumber people online.

Salesforce Introduces AIforce to Turn Its CRM Into an AI-Native Interface Layer

AIforce forms a new layer in Salesforce’s broader “Agentic Enterprise” architecture.

The Left ShiftThe Left Shift Bureau

Signoff Launches Agentic AI Platform, Deploys Across Nine Juniper Hotels

The platform connects data from areas including finance, operations, customer engagement, market intelligence and productivity system.

The Left ShiftThe Left Shift Bureau