Two of the world's top AI startups are ignoring requests from media publishers to stop scraping web content for free model training data, Business Insider has learned.
OpenAI and Anthropic were found to ignore or circumvent established web rules, known as robots.txt, that prevent the automated scraping of websites.
TollBit, a startup that aims to broker paid licensing deals between publishers and AI companies, found several AI companies engaging in this behavior and notified some major publishers in a letter on Friday that was previously reported by Reuters. The letter did not name the AI companies that allegedly circumvented the rules.
OpenAI and Anthropic have publicly stated that they respect robots.txt and block two specific web crawlers: GPTBot and ClaudeBot.
However, TollBit's findings show that such blocks are not being respected as claimed, with AI companies such as OpenAI and Anthropic simply choosing to “bypass” robots.txt in order to retrieve or scrape all content from a given website or page.
An OpenAI spokesperson declined to comment beyond pointing BI to a May company blog post in which the company said it takes web crawler permissions into account each time it trains a new model. An Anthropic spokesperson did not respond to an email seeking comment.
Robots.txt is a single piece of code that has been used since the late 1990s as a way for websites to tell bot crawlers that they don't want their data scraped or collected. It was widely accepted as one of the informal rules that support the web.
The rise of generative AI has startups and tech companies racing to build the most powerful AI models. A key ingredient is high-quality data. This thirst for training data is undermining robots.txt and the informal agreements that support the use of this code.
OpenAI develops the popular chatbot ChatGPT. The company's largest investor is Microsoft. Anthropic develops another relatively popular chatbot Claude. The company's largest investor is Amazon.
Both chatbots respond to users' questions in a human-like manner, and can do so because the AI models they are based on contain large amounts of text and data collected from the web, much of which is copyrighted or owned by its creators.
Last year, several technology companies filed a lawsuit with the U.S. Copyright Office arguing that nothing on the web should be considered copyrightable when it comes to AI training data.
OpenAI has signed deals with several publishers, including BI-owner Axel Springer, for access to their content, and the U.S. Copyright Office is expected to update its guidelines on AI and copyright later this year.
Are you a tech employee or someone with tips and insights to share? Contact Kali Hays. email address Or in a secure messaging appsignal +1-949-280-0267. Please contact us using a non-work device.
