Cloudflare's new AI crawler rules: stay found, keep control

On 15 September 2026 Cloudflare changed how its AI crawler controls work. Until then, a site owner who wanted to stop AI companies training on their content had one blunt tool, and using it could switch off search as well. The new setting separates the two, so you can refuse training and still be read by the search and answer engines your buyers use.

Our own view is simpler. If your website exists to win customers, let the bots in, and spend the time making sure nothing on the site is keeping them out by accident. A robots file that blocks AI crawlers is the most common problem we find.

What did Cloudflare change?

Cloudflare now sorts crawlers by why they're visiting as well as by who sent them. Every bot has one of three purposes:

• Search: building an index so pages can be found and quoted in results, AI answers included.
• Training: collecting content to build or fine-tune an AI model.
• Agent: fetching a page because a person has just asked an assistant to do something.

On top of that sits a new option, Disallow AI Training. It blocks the crawlers that collect content for training and lets the big search crawlers carry on indexing your site. Every Cloudflare plan has it, in each domain's security settings.

Why was the old approach a problem?

The biggest crawlers do two jobs at once. Googlebot, Bingbot and Applebot each index the web for search and also feed their company's AI work, and Cloudflare calls these mixed-use crawlers. Blocking one meant losing both jobs.

So site owners had a bad choice: give their content away for training, or drop out of search. Cloudflare's own figures show how most of them settled it. Fewer than 1% of sites on its network block search bots, but 17% had switched on some way of blocking AI training.

Those blunt tools had side effects that were easy to miss. A "block AI" toggle, or a robots.txt rule copied from somewhere else, could stop the assistants you want recommending you along with the ones building models. A blocked AI crawler doesn't show up as an error or a drop in rankings, and nothing in your analytics flags it. Usually the first sign is that assistants stop naming you.

What does "accountable" mean here?

Cloudflare only lets a mixed-use crawler through Disallow AI Training if its operator agrees to four things:

1. A way to opt out of training, through robots.txt or an equivalent.
2. A way to opt out of AI summaries of your pages.
3. Visibility, down to individual URLs, of how your content is used in search and training.
4. A commitment that opting out of training will not hurt your search rankings.

Apple, Google and Microsoft are named as accountable operators of mixed-use crawlers. Amazon, Anthropic, Meta and OpenAI are named for running separate crawlers for separate jobs, which is the arrangement Cloudflare would like everyone to reach.

Today the opt-outs look like this:

• Google: add a rule for Google-Extended in robots.txt. Google confirms this does not affect search ranking.
• Apple: add a rule for Applebot-Extended in robots.txt.
• Microsoft: use the NOARCHIVE meta tag or the Block URLs tool in Bing Webmaster Tools, with robots.txt support planned for early 2027.

Cloudflare also says it will publish, through Cloudflare Radar, which operators are keeping their commitments.

What happens to my existing settings?

For most sites, nothing you need to act on. Existing customers are being moved across automatically, and the effect is meant to stay the same. A training setting of Block or Block on pages with ads becomes Disallow AI Training, and a site that blocked everything, search included, stays blocked.

New domains added from 15 September get defaults. Sites without advertising start with search, training and agents all allowed. Sites that carry ads start with training disallowed and agents blocked on pages with ads.

Take a look anyway if someone set this up a year ago and has since moved on.

Why is being crawled good for AEO?

Most AI answers about buying decisions are grounded in a live search, so if the search crawler never read your pricing page or your comparison guide, the assistant has nothing of yours to quote and will quote a competitor. When a buyer asks an assistant to "look at their website and tell me if they do X", an agent comes to your site, and if it's blocked the assistant reports back that it couldn't find out. And training shapes what a model knows before it searches at all: which companies exist in a category, and what each is known for.

Cloudflare cites Pew Research finding that more than half of consumers read AI summaries in search, and that they are over 40% more likely to end their search after reading one. It also cites SimilarWeb and QuickSEO data showing that visitors referred by AI search convert at three to five times the rate of visitors from traditional search.

So should I disallow AI training or not?

If you sell a product or service and your content is there to win customers, leave everything allowed. Your articles are marketing, and you want as many engines as possible to know them and point people to you.

If the content is the product, as it is for a publisher or a paid research firm, Disallow AI Training is a good new option. You keep your search traffic and stop your work being absorbed into a model for free.

What should I check this week?

1. Open your Cloudflare security settings and confirm what each domain is set to for search, training and agents.
2. Read your robots.txt. Look for rules aimed at GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended or Applebot-Extended, and make sure each one is there on purpose.
3. Check your CDN or firewall rules for bot-blocking that sits outside Cloudflare's crawler setting, such as a challenge for every unknown user agent.
4. Ask the engines. Put your buyers' questions to ChatGPT, Gemini, Claude and Google and see whether you're named.

Firewall blocks are the hardest of these to spot, because the page often still answers. A bot challenge can come back looking like a normal response, with a "Just a moment" script where your content should be. Vizebel's technical review requests your homepage as each of the main search crawlers and as the ChatGPT, Claude and Perplexity agents, and reports any that are blocked or challenged. It tries the agents against one content page too, since Cloudflare's default for sites with ads only blocks them on pages that carry ads. A request from our servers isn't quite the same as one from OpenAI's, so the review also gives you the exact commands to run the same test yourself.