The presence of AI bots on news websites has been a topic of interest in recent years. Data journalist Ben Welsh conducted an analysis that revealed interesting insights about how these bots are being treated by news outlets.
According to Welsh’s analysis, just over a quarter of the news websites he surveyed are blocking Applebot-Extended. This indicates that a significant portion of news websites are wary of this particular AI bot. In comparison, a higher percentage of news websites are blocking OpenAI’s bot, showing a preference for Applebot-Extended.
Google introduced its own AI-specific bot, Google-Extended, which is blocked by nearly 43 percent of the news websites surveyed by Welsh. This suggests that Applebot-Extended may still be flying under the radar in comparison to Google’s bot. However, Welsh notes that the number of websites blocking Applebot-Extended is gradually increasing.
Welsh highlights a divide among news publishers regarding their approach to blocking AI bots. While some websites choose to block these bots, others make licensing deals with the bot owners in exchange for access. This decision-making process is not always clear, as some news organizations prefer to keep their reasons private.
Several major publishers have entered into strategic partnerships with AI bot owners like OpenAI and Perplexity. These partnerships often involve a business strategy, such as withholding data until an agreement is reached. The involvement of large publishers in such partnerships indicates a shift towards a more strategic approach in dealing with AI bots.
Maintaining an up-to-date block list for AI web-crawling bots can be challenging. Robots.txt files need to be edited manually, and with new AI agents constantly debuting, it can be difficult for publishers to keep track of which bots to block. Services like Dark Visitors offer solutions to automatically update a site’s robots.txt, particularly catering to publishers concerned about copyright issues.
Some CEOs from major media companies are directly involved in deciding which bots to block on their websites. This indicates the growing importance of AI bot management for digital publishers. Some outlets explicitly mention blocking AI scraping tools due to the lack of commercial agreements with the bot owners, highlighting the significance of partnerships in this space.
The presence of AI bots on news websites has become a complex issue, with news publishers having to make strategic decisions regarding blocking, licensing, and partnerships. As the AI landscape continues to evolve, it will be interesting to see how the relationship between news outlets and AI bot owners develops in the future.
Leave a Reply
You must be logged in to post a comment.