AI crawler wars threaten to make the web more closed for everyone

Hi folks, Paige Bailey here from the Google DeepMind dev rel team. For our developers out there, we know there's a constant trade off between model intelligence, speed and cost. Gemini 2.5 flash aims right at that challenge. It's got the speed you expect from Flash, but with upgraded reasoning power. And crucially, we've added controls like setting thinking budgets so you can decide how much reasoning to apply, optimizing for latency and costs. It's basically brains on a budget. So try out gemini 2.5 flash@aistudio.google.com and let us know what you built.

Welcome to MIT Technology Review Narrated My name is Matt Honan. I'm our Editor in Chief. Every week we'll bring you a fascinating new in depth story from the leading edge of science and technology, covering topics like AI, biotech, climate, energy, robotics and more. Here's this week's story. I hope you enjoy it.

Narrated by NOAA Listen to more of the best articles from the world's biggest publishers on the NOAA app or@newsoveraudio.com Shane Longpreh AI crawler wars threaten to make the web more closed for everyone we often take the Internet for granted. It's an ocean of information at our fingertips, and it simply works. But this system relies on swarms of crawlers. Bots that roam the web, visit millions of websites every day and report what they see. This is how Google powers its search engines, how Amazon sets competitive prices, and how Kayak aggregates travel listings. Beyond the world of commerce, crawlers are essential for monitoring web security, enabling accessibility tools, and preserving historical archives. Academics, journalists, and civil societies also rely on them to conduct crucial investigative research. Crawlers are endemic now, representing half of all Internet traffic, they will soon outpace human traffic. This unseen subway of the web ferries information from site to site day and night, and as of late, they serve one more purpose. Companies such as OpenAI use web crawled data to train their artificial intelligence systems like ChatGPT. Understandably, websites are now fighting back for fear that this invasive species, AI crawlers, will help displace the but there's a problem. This pushback is also threatening the transparency and open borders of the web that allow non AI applications to flourish. Unless we're thoughtful about how we fix this, the web will increasingly be fortified with logins, paywalls, and access tools that inhibit not just AI, but the biodiversity of real users and useful crawlers. To grasp the problem, it's important to understand how the web worked until recently, when crawlers and websites operated together in relative Symbiosis Crawlers were largely undisruptive and could even be beneficial, bringing people to websites from search engines like Google or Bing in exchange for their data. In turn, websites imposed few restrictions on crawlers, even helping them navigate their sites. Websites then and now use machine readable files called robots txt files to specify what content they wanted crawlers to leave alone. But there were few efforts to enforce these rules or identify crawlers that ignored them. The stakes seemed low, so sites didn't invest in obstructing those crawlers. But now the popularity of AI has thrown the crawler ecosystem into disarray. As with an invasive species, crawlers for AI have an insatiable and undiscerning appetite for data, hoovering up Wikipedia articles, academic papers and posts on Reddit review websites and blogs. All forms of data are on the menu, text, tables, images, audio and video. And the AI systems that result can, but not always, will be used in ways that compete directly with their sources of data. News sites fear AI chatbots will lure away their readers. Artists and designers fear that AI image generators will seduce their clients. And coding forums fear that AI code generators will supplant their contributors. In response, websites are starting to turn crawlers away at the door. The motivator is largely the same. AI systems and the crawlers that power them may undercut the economic interests of anyone who publishes content to the web by using the website site's own data. This realization has ignited a series of crawler wars rippling beneath the surface. Web publishers have responded to AI with a trifecta of lawsuits, legislation and computer science. What began with a litany of copyright infringement suits, including one from the New York Times, has turned into a wave of restrictions on use of websites data as well as legislation such as the EU AI act to protect copyright holders ability to opt out of AI training. However, legal and legislative verdicts could take years, while the consequences of AI adoption are immediate. So in the meantime, data creators have focused on tightening the data foreset at the source. Web crawlers. Since mid-2023, websites have erected crawler restrictions to over 25% of the highest quality data. Yet many of these restrictions can be simply ignored. And while major AI developers like OpenAI and Anthropic do claim to respect websites restrictions, they've been accused of ignoring them or aggressively overwhelming websites. The major technical support forum Ifixit is among those making such allegations. Now websites are turning to their last alternative anti crawling technologies. A plethora of new startups, tollbit, scalepost, etc. And web infrastructure companies like Cloudflare, estimated to support 20% of global web traffic, have begun to offer tools to detect, block and charge non human traffic. These tools erect obstacles that make sites harder to navigate or require crawlers to register. These measures still offer immediate protection. After all, AI companies can't use what they can't obtain, regardless of how courts rule on copyright and fair use. But the effect is that large web publishers, forums and sites are often raising the drawbridge to all crawlers, even those that pose no threat. This is even the case once they ink lucrative deals with AI companies that want to preserve exclusivity over that data. Ultimately, the web is being subdivided into territories where fewer crawlers are welcome. As this cat and mouse game accelerates, big players tend to outlast little ones. Large websites and publishers will defend their content in court or negotiate contracts. And massive tech companies can afford to license large data sets or create powerful crawlers to circumvent restrictions. But small creators such as visual artists, YouTube educators or bloggers may feel they have only two options hide their content behind logins and paywalls, or take it offline entirely. For real users, this is making it harder to access news articles, see content from their favourite creators, and navigate the web without hitting logins, subscription demands and captchas each step of the way Perhaps more concerning is the way large exclusive contracts with AI companies are subdividing the web. Each deal raises the website's incentive to remain exclusive and block anyone else from accessing the data, competitor or not. This will likely lead to further concentration of power in the hands of fewer AI developers and data publishers. A future where only large companies can license or crawl critical web data would suppress competition and fail to serve real users or many of them copyright holders. Put simply, following this path will shrink the biodiversity of the web. Crawlers from academic researchers, journalists and non AI applications may increasingly be denied open access. Unless we can nurture an ecosystem with different rules for different data uses, we may end up with strict borders across the web, exacting a price on openness and transparency. While this path is not easily avoided, defenders of the open Internet can insist on laws, policies and technical infrastructure that explicitly protect non competing uses of web data from exclusive contracts while still protecting data creators and publishers. These rights are not at odds. We have so much to lose or gain from the fight to get data access right across the Internet. As websites look for ways to adapt, we mustn't sacrifice the open Web on the altar of commercial AI. You are listening to MIT Technology Review, where Shane Longpre writes AI Crawler wars threaten to make the web more closed for everyone. This article was published on 11th February 2025 and was read by Les Smith for NOA.

Hi folks, Paige Bailey here from the Google DeepMind Dev Rel team. For our developers out there, we know there's a constant trade off between model intelligence, speed and cost. Gemini 2.5 Flash aims right at that challenge. It's got the speed you expect from Flash, but with upgraded reasoning power. And crucially, we've added controls like setting thinking budgets so you can decide how much reasoning to apply, optimizing for latency and costs. It's basically brains on a budget. So try out gemini 2.5 flash@aistudio.google.com and let us know what you built.