Reddit to Update Web Standard to Block Automated Data Scraping From Its Website

Reddit said that researchers and organizations such as the Internet Archive will continue to have access to its content for non-commercial use.

By Reuters | Updated: 26 June 2024 15:51 IST

Highlights

AI startups have reportedly been bypassing rules to gather content
Reddit said that it would update the Robots Exclusion Protocol
The platform will also block unknown bots and crawlers from data scraping

Reddit to Update Web Standard to Block Automated Data Scraping From Its Website

AI firms have been accused of plagiarizing content from publishers

Photo Credit: Reuters

Social media platform Reddit said on Tuesday it will update a Web standard used by the platform to block automated data scraping from its website, following reports that AI startups were bypassing the rule to gather content for their systems.

The move comes at a time when artificial intelligence firms have been accused of plagiarizing content from publishers to create AI-generated summaries without giving credit or asking for permission.

Reddit said that it would update the Robots Exclusion Protocol, or "robots.txt," a widely accepted standard meant to determine which parts of a site are allowed to be crawled.

Google Signs Licensing Deal to Train Its AI Models on Reddit Content

The company also said it will maintain rate-limiting, a technique used to control the number of requests from one particular entity, and will block unknown bots and crawlers from data scraping - collecting and saving raw information - on its website.

More recently, robots.txt has become a key tool that publishers employ to prevent tech companies from using their content free-of-charge to train AI algorithms and create summaries in response to some search queries.

Last week, a letter to publishers by the content licensing startup TollBit said that several AI firms were circumventing the web standard to scrape publisher sites.

Reddit Says 80 Percent Top Subreddits Open After Protest Blackout

This follows a Wired investigation which found that AI search startup Perplexity likely bypassed efforts to block its Web crawler via robots.txt.

Earlier in June, business media publisher Forbes accused Perplexity of plagiarizing its investigative stories for use in generative AI systems without giving credit.

Reddit said on Tuesday that researchers and organizations such as the Internet Archive will continue to have access to its content for non-commercial use.

Is the Samsung Galaxy Z Flip 5 the best foldable phone you can buy in India right now? We discuss the company's new clamshell-style foldable handset on the latest episode of Orbital, the Gadgets 360 podcast. Orbital is available on Spotify, Gaana, JioSaavn, Google Podcasts, Apple Podcasts, Amazon Music and wherever you get your podcasts.

Affiliate links may be automatically generated - see our ethics statement for details.

For the latest tech news and reviews, follow Gadgets 360 on X, Facebook, WhatsApp, Threads and Google News. For the latest videos on gadgets and tech, subscribe to our YouTube channel. If you want to know everything about top influencers, follow our in-house Who'sThat360 on Instagram and YouTube.