Why Websites Block AI Agents — and What Brands Should Do

More and more websites are blocking AI agents and crawlers. Security is usually the stated reason. The economic reason is simpler: agents do not see ads or buy subscriptions — and that is what much of the open web lives on.

---

1) The situation

Cloudflare describes it this way: more than half of internet traffic is now non-human, and daily requests from AI agents on its network have grown sharply over the past year. Agents do not respond to ads, but there is usually a person with a task behind them (Cloudflare).

The old deal — search engines crawl, visitors come back — no longer holds when an answer engine reads the page and returns only a summary.

---

2) Site owners' tools

• Block: one-click blocking of AI bots has existed since 2025 (Cloudflare).
• Disallow training, allow search: a new setting is meant to let mixed-use crawlers index for search while refusing training; according to Cloudflare, Apple, Google and Microsoft honour it or have committed to (Cloudflare).
• Charge: with pay per crawl a crawler pays per request — or receives "402 Payment Required" with a price. The feature is in closed beta (Cloudflare Docs).

---

3) The question for brands: block or open?

For publishers, blocking is often rational. For brands that want to sell, it is usually a mistake: if you lock agents out, you are not considered when agents research and buy — see agentic commerce and AEO.

A sensible split:

1. Open product, price and service information. That is what an agent needs to recommend you.
2. Protect or price exclusive editorial content. Studies, data and analysis that are your actual asset.
3. Decide training and search separately. Not a blanket "block AI".
4. Review regularly who comes. Which agents request, and what do they fetch?

---

Conclusion

Blocking agents is mainly a dispute about revenue models. Brands should not adopt it blindly: the content you sell with should be readable by agents. Protect what you want to be paid for.