Reddit has recently taken steps to block many of its search results from appearing on search engines other than Google. This move comes in the wake of a significant deal between Google and Reddit and aims to address concerns about AI misuse of Reddit’s content.
Google’s Deal with Reddit
In February, Google announced a new partnership with Reddit, allowing Google to use Reddit data to train its AI models. This deal, reportedly worth around $60 million, also ensures that Reddit results are more prominently featured in Google Search. Since the agreement, Reddit posts have started to outrank the websites they link to within Google’s search results.
Blocking Other Search Engines
However, Reddit has now updated its robots.txt file to block all bots from scraping its site, effectively preventing other search engines from displaying proper results from Reddit. This change was first reported by 404 Media, which noted the significant impact on search engines like Bing, DuckDuckGo, Mojeek, and Qwant. These platforms now struggle to show recent or complete results from Reddit.
Reason Behind the Block
Reddit explained that the change was driven by an increase in commercial entities scraping Reddit’s content for various use cases, including AI training. While not explicitly stated, it is evident that preventing AI training misuse is a primary concern for Reddit. A Reddit representative clarified that these issues with other search engines are unrelated to the Google partnership and are instead due to the updated robots.txt file targeting all crawlers unwilling to refrain from using Reddit data for AI training.
Reddit’s Willingness to Collaborate
Reddit is open to working with other entities on data crawling and is currently in discussions with multiple search engines. However, agreements have not been reached with all of them due to unresolved promises regarding the use of Reddit content, particularly for AI training. Despite these challenges, services like the Internet Archive and reddit4research continue to function as they comply with Reddit’s terms.
Current State of Search Engines
Paid search engine Kagi is still showing Reddit data, but only because it purchases some of its search index from Google, which retains access to Reddit data through the established deal. Meanwhile, other search engines remain affected by the block, leading to limited visibility of Reddit content on their platforms.
Conclusion
Reddit’s recent changes highlight the platform’s effort to control the use of its data, especially concerning AI training. While Google continues to benefit from its exclusive deal, other search engines are left negotiating to regain access. For users, this means relying on Google for the most up-to-date Reddit content until further agreements are reached.
