Cloudflare Rolls Out 'AI Index' in Private Beta to Empower Site Owners and AI Builders

Cloudflare today announced the private beta launch of AI Index—a tool that automatically builds an AI-optimised search index for domains on its network. With this, site operators gain control over how their content is made discoverable to AI systems, while developers access the data through standardised APIs.

Earlier this year, Cloudflare became the first internet infrastructure provider to block AI crawlers by default — unless explicit permission or compensation is given by content owners.

When enabled, AI Index creates and manages an internal search index and exposes tools like a search API, MCP server, and LLMs.txt files. These features allow domain owners to decide what content is included or excluded and who may access it. In addition, publishers can monetize access through Pay-per-crawl and x402 integrations.

For AI platforms and developers, Cloudflare offers a permissioned pub/sub mechanism: instead of blind crawling, they can subscribe to updates from participating domains.

When content changes, AI builders receive structured updates in real-time, reducing redundant crawling and improving data freshness. On a broader scale, Cloudflare will also offer an Open Index, which aggregates opt-in domain indexes into a unified search layer.

This initiative aims to rebalance power between content creators and AI companies. Cloudflare’s model grants creators ownership and compensation and encourages ethical access patterns—while giving AI developers cleaner, permissioned, and scalable infrastructure for content integration.

Last m0nth, Cloudflare also announced a major expansion to its Cloudflare One Zero Trust platform, unveiling new features aimed at helping businesses securely adopt and manage generative AI applications.

Recently, it also announced the private beta of Cloudflare Email Service, combining email sending and routing in a single developer-centric platform.