CloudFlare launches a market where websites can claim AI bots for scraping

CloudFlare, a cloud infrastructure provider that serves 20% of the web, announced on Tuesday the launch of a new market that rethinks the relationship between website owners and AI companies.

Last year, CloudFlare launched a tool for publishers to address the rise of ramp extensions in AI crawlers. One-click solution to block all AI botsas well as a Dashboard showing how AI crawlers access the site. In an interview in 2024, CloudFlare CEO Matthew Prince told TechCrunch that these products lay the foundation for a new type of market where publishers can distribute content to AI companies and compensate for it.

CloudFlare is now making its market a reality.

It’s called Payment Per Crawl, and CloudFlare will begin “experiment” in private beta on Tuesday. The owner of the experimental website can choose to scrape the site off the AI ​​crawler individually, at a set rate. This is all “crawl” micropayments. Alternatively, website owners can choose to have AI crawlers scrape the site for free or block it entirely. CloudFlare claims that the tool allows website owners to check whether Crawler is scraping the site for AI training data, whether it appears in AI search responses, or for other purposes.

On a large scale, CloudFlare’s Marketplace is a big idea that can provide publishers with potential business models for the AI ​​era. It also puts CloudFlare at the heart of everything. The launch of the market comes when news publishers face existential questions about how to reach their readers as Google’s search traffic disappears and AI chatbots become more popular.

There is no clear answer as to how news publishers will survive in the age of AI. Some of the new York Times have submitted Litigation against high-tech companies To train AI models with news articles without permission. On the other hand, other publishers have it Attacked multi-year deals to get licensed for content For AI model training, and for viewing content in AI chatbot responses.

Still, only large publishers have attacked AI licensing transactions, and it is still unclear whether they will provide a meaningful revenue stream. CloudFlare aims to create more durable systems that allow publishers to price their own terms.

The company announced Tuesday that a new website configured using CloudFlare will block all AI crawlers by default. Site owners must grant permission to access the site of a particular AI crawler. It says that with the CloudFlare change, all new domains give “default controls”.

Several large publishers, including Conde Nast, Time, The Associated Press, Atlantic, Adweek and Fortune, have signed on with CloudFlare to block AI crawling by default to support the company’s broader goal of “a permission-based approach to crawling.”

The business models that many of these publishers relied on for decades have slowly become unreliable. Historically, online publishers have allowed Google to scrape the site in exchange for referrals of Google searches translated into traffic to the site.

However, new data from CloudFlare suggests that Publishers may be worse in the AI ​​era than in the Google search era. meanwhile Some websites cite ChatGpt as the main traffic sourceit seems mostly not a case.

In June this year, CloudFlare says it found that every time Google’s Crawler introduced it, it had shattered its website 14 times. Meanwhile, Openai’s Crawler cut 17,000 websites for each referral, but humanity cut 73,000 times for each referral.

Meanwhile, Openai and Google are Visit the website on your behalfcollects information and responds directly to the user. The future where these tools are mainstream will have a major impact on publishers who rely on readers visiting the site.

CloudFlare points out that “true possibilities” of wages per crawl could emerge in the future of “agents.”

“What if the agent paywall can work completely programmatically at the network edge? Ask your favorite deep research program that will help you integrate the latest cancer research and legal briefs, or help you find the best restaurants in Soho and give that agent the budget to spend to get the best relevant content.”

To participate in the CloudFlare experimental market, both AI companies and publishers must set up with their CloudFlare accounts. With the account, both parties can set a price they wish to buy and sell “crawls” of publisher content. CloudFlare acts as an intermediary in these transactions, billing AI companies and distributing revenues to publishers.

CloudFlare spokesman Ripley Park tells TechCrunch that there are no stubcoins or cryptocurrencies related to salary per crawl at this point, despite many suggestions. Digital currency is best for things like this.

CloudFlare’s marketplace feels like a bold vision for the future where many publishers and AI companies need to board. Still, there is no guarantee that the publisher will get a fair deal. Given that they are currently rubbing content for free, it can be difficult to convince AI companies to take part.

Nonetheless, CloudFlare appears to be one of the few companies in a position to make such a market a reality.