Cloudflare Offers Control to Website Owners for AI-Based Bots on 20% of the Internet

Cloudflare, who process requests for over 20% of the world’s websites, has announced a new AI bot control system. They’re introducing fine grained controls on bots for the first time, including Search bots, Agent bots (that take action on a user’s behalf) and Training bots (that harvest data to train AI models). Previously, you could only allow all bots or block all bots.

1New Rules Starting September 15, 2026

From September 15th 2026, Cloudflare will begin default blocking Training and Agent bots on sites with advertising, but allowing Search bots. The feature is available to all users, not just their Enterprise plan. There’s also a feature called BotBase for Enterprise users which provides access to a database of every known bot.

2Transparency as a Condition of Access

Their plans include providing Verified status to bot developers. To qualify, they have to adhere to strict behavioural standards and disclosure requirements. If they lose this status, they will be blocked for over 20% of the internet’s domains — which makes Cloudflare’s policy a key bedrock of the industry. This is the first structural impediment that AI developers, who have quietly amassed data for years, have faced.