Cloudflare launches marketplace for AI companies that want to scrape websites

Cloudflare, one of the biggest network operators in the world, has announced an online marketplace where AI companies can pay website owners for permission to scrape the content of their websites.

Over the past 30 years, the internet and the way content is being searched and displayed have changed drastically.

In the early days, search engines used so-called ‘spiders’ to crawl websites to offer internet users the best content that was available at the time. These top results were then listed, containing ten blue links on a search engine result page or SERP.

This worked for decades because Google offered content creators something in return: traffic to their websites.

Because of this traffic, website owners could derive value from their content in several ways. They could earn revenue by displaying ads on their sites, by selling subscriptions, or simply by having the satisfaction that there were people out there enjoying their content.

The rise of artificial intelligence (AI) has utterly changed the relationship between Google and content creators. Instead of referring users to websites, Google now offers ready-made answers with a search box and AI Overviews, meaning that users no longer have to leave Google’s SERP to find answers to their questions.

Google Search users may like this development, but content creators are feeling the pain because of the decreasing traffic numbers. As a matter of fact, it’s become way more difficult for website owners to get the same volume of traffic and to generate revenue. This is due to AI crawlers that scrape the internet and offer content creators nothing in return.

“The problem is whether you create content to sell ads, sell subscriptions, or just to know that people value what you’ve created, an AI-driven web doesn’t reward content creators the way that the old search-driven web did. And that means the deal that Google made to take content in exchange for sending you traffic just doesn’t make sense anymore,” Matthew Prince, Co-Founder and CEO of Cloudflare, writes in a blog post.

He promises a solution to this problem: an online marketplace where content creators and AI companies can come together and make arrangements about crawling websites. Website owners can choose to let AI crawlers scrape their site at a set rate (a micropayment for every single crawl of ‘Pay per Crawl’), to have AI crawlers scrape their site for free, or to have them blocked altogether.

At the same time, Cloudflare is blocking AI crawlers by default.

“We believe that if we can begin to score and value content not on how much traffic it generates, but on how much it furthers knowledge, we not only will help AI engines get better faster, but also potentially facilitate a new golden age of high-value content creation,” Prince concludes.

The online marketplace and ‘Pay per Crawl’ are currently in private beta. It’s unclear when they will be publicly available for content creators and AI companies.