Cloudflare: ‘Perplexity’s AI bots ignore no-crawl directives’
Perplexity’s AI bots crawl and scrape content from websites, but when the AI bots come across a network block, they obscure their crawling identity to circumvent the website’s preferences.
In a lengthy blog post, internet service provider Cloudflare says it has received several complaints from its customers. They claim that Perplexity’s web crawlers can crawl and scrape their website, even when they specifically disallow or block Perplexity in their robots.txt file or firewall rules.
Cloudflare tested this by purchasing and launching several new domains and then blocking Perplexity’s web crawlers. According to the tech company, these domains were newly registered and had not been previously indexed by search engines or made publicly accessible.
The websites were then provided with a robots.txt file, which prohibited all crawlers from visiting them. Next, Cloudflare asked Perplexity’s AI bots questions about the domains to see what would happen. Despite the ban, the AI bots were still providing detailed information about the content on each of the restricted domains.
Cloudflare discovered that Perplexity initially used its well-known public web crawlers: PerplexityBot and Perplexity-User. If these are blocked, Perplexity uses a generic browser intended to impersonate Google Chrome on macOS when its declared crawler is blocked.
“This undeclared crawler utilized multiple IPs not listed in Perplexity’s official IP range, and would rotate through these IPs in response to the restrictive robots.txt policy and block from Cloudflare,” researchers from Cloudflare found out. This was observed across tens of thousands of domains and millions of requests per day.
Perplexity says Cloudflare’s report is nothing more than a “sales pitch” and full of “misunderstandings.” The AI startup claims its bots aren’t crawlers but rather agents that retrieve information on behalf of users and thus function as digital assistants. This justifies the bots’ behavior, the company says.
Perplexity doesn’t elaborate on the detours these bots take to gain access to websites that block Perplexity.
This is not the first time Perplexity has been accused of secretly scraping content from websites. Last year, the company was accused by news outlets like Wired of plagiarizing their content and circumventing paywalls. At the time, Perplexity’s CEO Aravind Srinivas said that this was due to third-party crawlers used by the company.