Cloudflare's tools enable websites to monitor and control how AI model providers access their content
Cloudflare has launched AI Audit, a new tool that empowers website owners to monitor, control, and potentially monetize how AI models interact with their content, aiming to rebalance the relationship between content creators and AI companies.
Cloudflare, the network and security products provider actively helping customers get bots and scrapers under control, has introduced AI Audit, a tool suite to empower websites to track and manage how AI model providers interact with their content. Additionally, Cloudflare shared it is experimenting with a new feature that will allow content creators to set a fair price for access to content that AI companies can use for model training and retrieval augmented generation purposes.
As Cloudflare notes in its press release, even when websites are aware that AI bots are using their content, there is no reliable way to indicate which kinds of content scanning are allowed or to take action when unwanted content scraping is happening. If the infamous incident wherein Perplexity AI was the subject of several accusations of unethical web scraping, is that many AI companies have too little regard for current standards, such as the Robots Exclusion Protocol, which, to date, stands as one of the only methods websites have to signal that the content in a webpage (technically) should not be scanned or scraped by web crawlers.
Whether we want to recognize it or not, this has led to a destruction of value for many businesses, especially those without sufficient resources, context, and power to strike a deal directly with AI model providers, just as many of the biggest media and publishing names have sought to do with OpenAI and others. The only way to address this issue, says Cloudflare, is to give the control back to the content creators. Thus, its AI Audit tools will allow websites, regardless of their size, to control all bots free of charge; harness advanced analytics indicating why, when, and how often their content is being accessed; establish terms of use that will legally protect their content in case of misuse; and, with the new experimental feature, set fair prices in exchange for the right to leverage the website's content.
Cloudflare's AI Audit tools are available in the AI tab on the Cloudflare dashboard for all customers starting Monday, September 23. Customers interested in trying the price-setting tool beta should join the waitlist.
