Cloudflare will block Training and Agent AI crawlers by default from September 15

Cloudflare will block Training and Agent AI crawlers by default on ad-monetized pages starting September 15, 2026, while continuing to allow Search crawlers.

The controls, rolled out July 1 to all customers including the free tier, replace a single “Block AI Bots” switch with separate handling for three categories of automated traffic: Search, Agent and Training crawlers. That lets site owners permit search indexing while refusing model-training scrapers, or the reverse.

There is a catch for multi-purpose bots. A crawler such as Googlebot will be blocked if any of its functions fall into a blocked category, which could cut a site off from services it wants alongside ones it does not. Enterprise customers also get BotBase, a searchable database of tracked and classified bots.

The move fits a broader fight over who may scrape the open web to train AI systems, and whether publishers can charge for or refuse that access. Cloudflare sits in front of a large share of internet traffic, so a default that blocks training crawlers on monetized pages shifts leverage toward site owners.

What the change does not do is settle enforcement. Crawlers can ignore or spoof directives, and the controls depend on Cloudflare’s ability to identify bots accurately. The September default is opt-out, so sites that take no action will inherit the new blocking behavior.