Open source developers are countering AI crawlers with ingenuity and determination

Open source devs fight AI crawlers with clever tools, blocking bots ignoring rules.

: The invasion of AI web-crawling bots is considered an internet nuisance, particularly by open source developers like Niccolò Venerandi, as they disproportionately affect sites hosting free and open source software. These bots often ignore the Robots Exclusion Protocol, causing issues like DDoS outages as highlighted by Xe Iaso’s encounter with AmazonBot. Iaso's response was to create Anubis, a tool providing proof-of-work checks to filter out human requests from bot traffic, a solution embraced by the development community. Other developers like Drew DeVault and Kevin Fenzi have also shared their struggles and proposed vengeance tactics, including using the Nepenthes tool and Cloudflare's AI Labyrinth to thwart these crawlers.

Open source developers are innovatively countering the pervasive issue of unauthorized AI bots that aggressively scrape their websites. They consider these bots as invasive 'cockroaches' of the internet, according to developer Niccolò Venerandi, who highlights how open source projects, due to their public sharing nature, are particularly vulnerable and lack the extensive resources commercial entities possess. Despite the existence of protocols intended to manage web-crawling bots' behavior, such as the Robots Exclusion Protocol, many AI bots ignore these instructions, exacerbating the problem for developers who often face severe disruptions, such as DDoS (Distributed Denial-of-Service) outages, as noted by FOSS developer Xe Iaso in a case with AmazonBot targeting a Git server website.

As a solution to combat these disruptive bots, Xe Iaso developed Anubis, named humorously after the Egyptian god associated with judgment, which functions as a reverse proxy proof-of-work check system. Anubis allows only genuine human user requests to pass through, blocking the bots and providing a charming touch by displaying an anime illustration upon successful verification. Open source developers have embraced this method eagerly, demonstrated by the swift success of Anubis on GitHub, earning numerous stars and contributors within days.

The stories of resistance don't stop there. Developers like Drew DeVault, the founder of SourceHut, have detailed their own challenges dealing with aggressive AI crawlers that require them to dedicate a substantial portion of their time to mitigating these threats. Kevin Fenzi, managing the Linux Fedora project, even resorted to the drastic measure of blocking an entire country's IP addresses due to incessant bot activity, reflecting the extreme actions some developers are forced into.

Alternative defensive strategies reflect a blend of creativity and humor. An unknown developer, “Aaron,” released Nepenthes, a tool designed to bait crawlers into labyrinthine paths of deceptive content, evoking a sense of retaliation. This innovative approach is not isolated; companies like Cloudflare have developed AI Labyrinth to deter these bots by serving them irrelevant data, conserving legitimate site data for genuine users.

Despite these creative solutions, there is clearly an underlying plea among developers for more systemic changes to stop the legitimization and proliferation of AI crawler technologies that they view as detrimental. Drew DeVault has been particularly vocal, urging a conscious move away from supporting AI-driven applications like LLMs, image generators, and others that are perceived negatively by the development community.

Sources: TechCrunch, Ars Technica, GitHub, Hacker News, Cloudflare blogs

technologySoftwareSecurity