Cloudflare Separates AI Training From Search, but Googlebot Rules Need Careful Review
Cloudflare's reworked AI traffic controls give site owners more choice over search, agents and training. Multi-purpose crawlers make configuration review essential.
Cloudflare has replaced its former one-click AI bot blocking approach with separate controls for Search, Agent and Training traffic. The change gives website owners more specific choices over how AI systems access their content, but it also creates an important operational issue: crawlers that serve both search and AI training purposes can be affected by Training rules. That includes Googlebot under Cloudflare's combined-crawler behavior.
Cloudflare deployed the reworked system on September 15, 2026. As detailed in Cloudflare's official announcement about ending the search-or-training tradeoff, the company is positioning the controls as a way for publishers and other site operators to allow search discovery without automatically allowing AI training.
For businesses that depend on organic search traffic, the practical lesson is straightforward: blocking AI training is no longer a single, isolated decision. Site owners need to check whether their chosen setting also affects a multi-purpose crawler they want to keep available for search indexing.
What Cloudflare's new AI traffic controls change
Cloudflare's revised model separates crawler access into three categories:
- Search covers crawlers used to discover and index content for search.
- Agent covers AI agents that access web content to complete tasks or retrieve information.
- Training covers crawlers that collect content for AI model training.
The categories replace the simpler "Block AI Bots" approach. Cloudflare says Search remains allowed by default for new domains, while Training and Agent crawlers are blocked by default on pages that display ads. Existing customers can opt out or adjust their settings.
This separation matters because search indexing and AI training are not always performed by wholly separate bots. Cloudflare identifies multi-purpose crawlers that combine Search and Training behavior as a key complication. When Training is blocked, the combined behavior can also block a crawler such as Googlebot.
| Cloudflare traffic category | Purpose described by Cloudflare | Relevant default or configuration detail |
|---|---|---|
| Search | Content discovery and search indexing | Allowed by default for new domains |
| Agent | AI agents accessing content for tasks or retrieval | Blocked by default on pages that display ads |
| Training | Collection of content for AI model training | Blocked by default on pages that display ads |
| Multi-purpose crawlers | Combined Search and Training behavior, including Googlebot | Can be blocked when Training is blocked |
The table is not a claim that every site will receive the same crawler treatment. Cloudflare's settings, a site's existing configuration and the crawler's combined behavior determine the outcome. The key distinction is that a setting intended to limit training can have consequences for search access when one crawler spans both purposes.
Why Googlebot access deserves attention
Googlebot is especially relevant because Google search indexing remains a major discovery channel for many websites. Under Cloudflare's new regime, access can be affected where the applicable rules classify the crawler through its combined Search and Training behavior.
Cloudflare provides a Disallow AI Training option intended to let operators preserve Search indexing while preventing Training. The research also identifies tokens such as Google-Extended as another way operators may preserve indexing while disallowing training. Which route is appropriate depends on the site's goals and its Cloudflare configuration.
This is a more nuanced choice than simply deciding whether a business wants its content used to train AI models. A publisher, ecommerce business or service company may want Google to index product, service or help pages while still limiting training-related collection. Conversely, a site that deliberately restricts all automated access may accept reduced search availability as part of that decision.
A practical configuration review for website owners
The update calls for a focused review rather than an automatic switch to the most restrictive setting. First, identify whether the site uses Cloudflare's AI traffic controls and whether it has an existing legacy configuration. Next, establish which forms of access matter most: search indexing, AI agent visits, AI training, or a combination.
Then review the selected controls against the potential effect on multi-purpose crawlers. Where search visibility is important, the Disallow AI Training option is the Cloudflare-provided path specifically described as preserving indexing while preventing training. Operators should also verify that the intended crawler access is working after a configuration change, rather than assuming that a broad block applies only to training traffic.
This is particularly important for teams that manage advertising-supported pages, because Cloudflare's September defaults block Training and Agent crawlers on pages that display ads. It is also relevant for existing Cloudflare customers, who may need to opt out or alter settings based on their own access policy.
More control over AI traffic and content value
The new controls are part of a wider Cloudflare effort to give publishers more information and leverage over AI-related crawling. The company also introduced enhancements including BotBase, Content Signals and an Attribution/Pay-Per-Use framework. Cloudflare describes these tools and frameworks as ways to understand AI traffic better and support content monetization.
For site owners, the immediate value is not a guarantee of revenue or AI search exposure. It is clearer control over different kinds of automated access. That distinction is increasingly important as web traffic includes traditional search crawlers, AI agents and systems collecting data for model development.
Search access and training permissions now need to be treated as separate website decisions. If your company depends on organic discovery, an AI search visibility audit can identify where crawler rules, content signals, and indexability may weaken visibility in AI-driven results. GEO Search Leads helps turn those findings into practical priorities for pages that need to remain discoverable while you control training access. Start an AI visibility scan today.
Frequently Asked Questions
What are Cloudflare's new AI traffic controls?
Cloudflare now separates automated traffic into Search, Agent and Training categories. The controls replace the older one-click "Block AI Bots" setting and let operators set access rules more specifically.
Can blocking AI training affect Googlebot indexing?
Yes. Cloudflare says multi-purpose crawlers that combine Search and Training behavior, including Googlebot, can be blocked when Training is blocked. Site owners should review their selected rules if Google indexing matters to them.
How can a site prevent AI training while keeping search indexing?
Cloudflare offers a Disallow AI Training option designed to allow Search indexing while preventing Training. The research also identifies tokens such as Google-Extended as another way to preserve indexing while disallowing training.
What are the default settings for new Cloudflare domains?
Cloudflare says Search is allowed by default for new domains. Training and Agent crawlers are blocked by default on pages that display ads.
Conclusion
Cloudflare's three-part AI traffic system provides a more precise alternative to an all-purpose AI bot block. Its most important practical consequence is that search indexing and AI training restrictions must be configured together, especially where a multi-purpose crawler such as Googlebot is involved. Website owners that want to protect search discovery while limiting AI training should review their current Cloudflare settings and use the option that matches both goals.