Cloudflare Wants AI Companies to Pay Publishers for What Their Content Powers
Charging AI crawlers to fetch a page was never the right question. Cloudflare's new model tries to charge for whether the content got used at all.
Publishers have spent two years watching AI companies crawl their websites for free, then answer the reader's question directly, with no click, no ad impression, and no payment ever reaching the publisher. Cloudflare thinks it has found the fix, and it is not the one it originally proposed.
From Pay Per Crawl to Pay Per Use
Cloudflare's first attempt at this problem let publishers charge AI crawlers a flat fee every time a bot fetched a page. It was a reasonable starting point and Cloudflare has since said plainly that charging per fetch is not enough. A crawler can fetch a page and never actually use it in an answer, or use a tiny fragment of it in millions of answers, and a flat per-crawl fee captures neither situation accurately.
The newer model, Pay Per Use, tries to charge for the thing that actually matters: whether the content shows up inside an AI-generated answer, not whether a bot happened to request it. Cloudflare's own framing is blunt about why this shift happened, the old model didn't reflect how AI answers actually consume content.
The bigger structural move
Alongside the payment model, Cloudflare is changing what happens by default. Starting later this year, crawlers that blend search, AI-agent and AI-training functions into one bot will be blocked by default on ad-supported pages for new domains and free-tier customers, unless a site owner explicitly opts back in. Pure search crawlers stay allowed. Everything blurrier than that now has to ask permission first.
The Ad Tribe angle
This is not yet the industry-wide fix it will eventually need to be. Only a couple of named partners are live on the new payment model, and the flat per-crawl option remains in limited beta rather than broad rollout. But the direction matters more than the current adoption numbers: the internet's biggest infrastructure company has publicly concluded that "AI crawled it" is the wrong question, and "did AI actually use it" is the right one. That is a much harder thing to measure, and a much fairer thing to pay for.
Advertisement
