Cloudflare Introduces AI Audit Tool to Block Data Scraping AI Bots from Accessing Websites

Cloudflare is offering AI Audit to all of its existing clients at no additional cost.

Written by Akash Dutta, Edited by David Delima | Updated: 24 September 2024 18:17 IST

Cloudflare Introduces AI Audit Tool to Block Data Scraping AI Bots from Accessing Websites

Photo Credit: Unsplash/Anna Shvets

Cloudflare is also developing a tool that will allow website owners to set a fair price on their content

Highlights

AI Audit is a set of tools that gives website owners control over content
Users will be able to selectively block and allow AI bots
AI Audit will also offer analytics about how bots are crawling websites

Cloudflare has unveiled AI Audit, a new suite of tools that are designed to offer website owners and bloggers more control over their content and how it is used by automated systems. The security tool will enable users to block or allow artificial intelligence (AI) bots which often crawl websites to scrape data. The company is offering this tool to all of its existing customers without any extra charge. Cloudflare is also adding a feature to let website owners see detailed analytics of which bots are visiting the platform and their behaviour patterns.

Cloudflare Introduces AI Audit Tool to Block AI Bots

With the increasing popularity of generative AI technology, there has been a rush to train large language models (LLMs) with human-created data. This is important not only to create a foundation model but also to advance and improve it. However, the main issue here is accessing publicly available data sources. Most AI firms have already gone through these datasets, and now require more data to train AI models.

AI bots are designed to help developers gather more information to train AI models. They can be understood as systematic programmes imitating a real user that can enter websites and copy text, image, and video data. These AI bots can scrape through large volumes of data in a short period and deliver it to the AI model. In recent times, several media firms and large websites have filed lawsuits against AI firms alleging plagiarism and illegally using data to feed LLMs.

Meta AI's Voice Mode May Sound Like John Cena and These Other Celebrities

Cloudflare's AI Audit tool comes as a protective layer that can block such bots from accessing websites. The company revealed in a press release that has also made improvements to the tool to give users more control over which bots will be restricted and which bots will be allowed access. This is useful in cases where the platform has struck a deal with an AI firm and does not mind its bots taking the data. Alternatively, the website owner might want to give access to certain AI models which attribute the source of the data to gain a better reach.

Cloudflare highlighted that it is also building a workflow where website owners can set a fair price on their content. AI bot owners, on the other hand, will be able to transact with this firewall and once the amount has been paid, they will be given the right to scan content. The company highlights that its marketplace-like tool can be beneficial for users who do not have the bandwidth or resources to negotiate and strike deals with each AI firm that approaches their website.

Comments

For the latest tech news and reviews, follow Gadgets 360 on X, Facebook, WhatsApp, Threads and Google News. For the latest videos on gadgets and tech, subscribe to our YouTube channel. If you want to know everything about top influencers, follow our in-house Who'sThat360 on Instagram and YouTube.

Further reading: Cloudflare, AI, Artificial Intelligence, Cybersecurity

Akash Dutta Email Akash Dutta

Akash Dutta is a Senior Sub Editor at Gadgets 360. He is particularly interested in the social impact of technological developments and loves reading about emerging fields such as AI, metaverse, and fediverse. In his free time, he can be seen supporting his favourite football club - Chelsea, watching movies and anime, and sharing passionate opinions on food. More