Cloudflare launches a marketplace that lets websites charge AI bots for scraping

Cloudflare's new marketplace shifts the AI-web scraping dynamics, empowering site owners.

: Cloudflare has introduced a marketplace called Pay per Crawl in private beta, empowering website owners by allowing them to charge AI bots for scraping their content. This initiative is part of Cloudflare's ongoing efforts to provide tools that give site owners control, such as blocking or monitoring AI bots' activities. Notable publishers, including Conde Nast and TIME, are supporting Cloudflare’s move to block AI crawlers by default, reflecting a shift from search engine traffic to AI interactions. Matthew Prince, CEO of Cloudflare, expressed ambitions to explore stablecoin options for these transactions in the future.

Cloudflare has launched a new marketplace called “Pay Per Crawl” that allows websites to charge AI bots for accessing their content. This initiative introduces a micropayment system where site owners can set individual fees for each AI crawler, offering a potential revenue stream for publishers whose content is often used without compensation.

Alongside this, Cloudflare will now block AI crawlers by default on all new domains unless site owners explicitly allow access. This marks a significant policy shift aimed at giving website administrators more control over how their data is used by AI firms and reducing unauthorized scraping.

The move has garnered support from major publishers such as Condé Nast, Time, The Atlantic, Reddit, and BuzzFeed, who are increasingly concerned about AI bots consuming their content without delivering meaningful traffic in return. These publishers are looking for ways to protect intellectual property and ensure fair use.

The marketplace uses standard web protocols like HTTP status codes and robots.txt files to manage bot permissions and payment expectations. AI companies are expected to register their crawlers, identify their use cases (training, inference, search), and participate in a more transparent and equitable ecosystem.

Although still in private beta, the service reflects Cloudflare’s push toward a “permission-based internet” where content access is conditional on terms set by creators. This model could become a foundational framework for managing data usage in the age of AI.

Sources: Business Insider, Reuters, The Verge, TechCrunch, Nieman Lab

AIwebCloud