Lately, a bunch of big news publishers have started blocking the Internet Archive's crawlers, the ones that feed the Wayback Machine with snapshots of web pages. This tool has been saving over a trillion pages since the mid-90s, helping journalists, researchers, and courts check original versions of stories that get edited or pulled later. The main worry? AI companies might sneak in through the archive to grab content for training models without permission, even if publishers block direct scrapers. Outlets like NYT added bots like archive.org_bot to their robots.txt files late last year, and now at least 23 major sites, including USA Today and The Guardian, do the same. An analysis of over 1,100 news sites found 241 blocking at least one Archive bot, mostly Gannett-owned ones.
Why's this happening? Publishers want to protect their intellectual property and stop AI from using their journalism to build competing tools. NYT says they value human-led reporting and need lawful access control. The Guardian limited article access after spotting the Archive as a top crawler in logs, fearing it as a backdoor. It's not just theory; some evidence shows AI firms have tapped archives before, though not always proven for these sites. On the flip side, critics like EFF argue this won't halt AI but erases web history, leaving gaps where quality news vanishes while junk sites stay archived. Internet Archive founder Brewster Kahle warns it limits public access to records, hurting efforts against info chaos.
The balance is tricky: publishers fight real revenue threats from AI summaries stealing clicks, but blocking a nonprofit library risks future proof of events. No easy fix yet, as talks between Archive and outlets continue amid growing blocks. This could mean biased historical records, with big names missing from snapshots.
