Imagine suddenly finding your institution's digital doors battered down by automated crawlers harvesting your collection at unimaginable speeds. Protecting archive and collection systems from the volume-driven impacts caused by AI-driven crawlers and automated scraping bots is now a serious problem for online collections. In the past few years, we’ve seen this play out many times across our clients' systems, with institutions that had run smoothly for years suddenly under attack from this new and overwhelming type of traffic.
While the way in which these attacks happen changes monthly, the foundation of the problem remains the same. You do not need a computer science degree to understand why these outages happen, nor what steps must be taken to stop them. Beyond the immediate technical defense issues, this escalation in automated traffic forces every public archive to ask an important philosophical question: What does true open access look like to us, and exactly how far do we allow it to stretch?
The Problem
As of mid-2026, Cloudflare Radar data shows that automated bots now generate over 57.5% of all internet traffic - so, the machines have overtaken human web browsing (and, according to the graph below from Cloudflare, they have for some time).
This is a big shift in internet demographics that is having a noticeable effect on your public archive, and it’s the reality of how the Internet is changing. Large Language Models (LLMs) and advanced search indices need vast amounts of high-quality, structured data to function.
While bots have been around on the Internet - doing their thing and behaving politely - for decades, the issue is their new volume and concurrency. A single AI crawler can open hundreds of simultaneous connections to a web server. When several bots arrive at once, or when a single aggressive crawler isn’t playing nicely, the server can become overloaded and unable to serve real human visitors. Response times climb, queues back up, and the site goes down, disrupting access for legitimate users.
Why archives?
Archive and collection sites are such rich targets for data harvesting. They hold large amounts of structured, well-labelled, unique, and authentic content.
For AI models to generate accurate and coherent responses, they need vast amounts of human-verified information to avoid the “garbage in, garbage out” trap. Unlike the chaotic, unstructured text found on much of the Internet, especially social media and public forums, institutional collections present this information in an extremely ordered way. Archival sites offer factual accuracy, transcriptions of non-born-digital content, and highly structured metadata, with every record typically tagged by professionals with authoritative dates, creators, locations, subject area, and more.
That is exactly what AI companies and data brokers desperately want for training their models. Additionally, public archives operate on a fundamental mandate of open access. They rarely hide their collections behind mandatory user accounts, paywalls, or geographic restrictions. To an automated web scraper, an open digital archive is essentially a highly curated, unguarded gold mine, served on a silver platter.
The result is chaotic and often disastrous for smaller organisations. While a niche historical manuscript might normally attract a handful of dedicated scholars over a year, an automated script will systematically download that document and the tens of thousands of others housed beside it in seconds, and continue doing so across the collection until the system can no longer keep up with the load. Consequently, these systems may attract a disproportionate volume of automated traffic relative to their human audience. We've seen client collections receive more automated requests in a single hour than they'd normally see from human visitors in a month.
Death by a Thousand Bots
The challenge we now face is a mismatch between how scrapers harvest data and how public archives have historically served it. Across the institutions we work with, this mismatch tends to show up in the same three ways.
Concurrency vs. Capacity
Human traffic to a digital archive is usually modest and highly predictable. It ebbs and flows naturally, following local time zones and the typical workweek, and sometimes peaks after genealogy TV shows are aired. To keep budgets down, archival servers are carefully provisioned for this steady, sustainable baseline. They are generally not built for an automated crawler attempting to download an entire century of catalogue records in three minutes. When a bot opens hundreds of simultaneous connections, the server is quickly overwhelmed and is unable to serve anyone else, leaving the legitimate users staring at a broken page.
Deep Traversal
By their very nature, archival URLs tend to be deeply nested and incredibly numerous. A single historical collection might contain thousands of cross-referenced items, subjects, and creators. When a human manages this, they follow a specific thread of inquiry, with time taken to read each new record they load. When a crawler arrives, it clicks every link available, which can easily generate millions of rapid-fire requests from a single initial URL. Because archival catalogues are so interconnected, oblivious bots often traverse every possible combination of filters and metadata - even administrative functions - dragging down the database in the process.
Heavy Assets
Many collections serve exceptionally large files, such as high-resolution master images, multi-page PDFs, videos, and oral history audio recordings. Every automated request for these assets can be highly expensive in terms of both network bandwidth and server CPU time. While a human might download a single large archival photograph to study its details, a scraper bot will blindly request thousands of them per minute. The archival sites’ servers end up consuming massive amounts of computing power and bandwidth to deliver large files to machines that will never actually look at them.
Data Risks
The threat extends beyond outages and increased computing costs. When automated systems harvest archival records at scale, they strip away the context and access protocols that institutions rely on to control their collections.
Unauthorised Reuse and Copyright
Archives frequently host materials under licensing agreements, donor restrictions, or traditional knowledge labels. A human researcher handles these restrictions by submitting reading room requests, following the terms and conditions, or submitting permission requests. A crawler often doesn’t follow the same processes. It just pulls the raw data and ingests it. This extraction often strips the original authoritative dates, creators, locations, and subject matter and blatantly ignores the conditional permissions under which the institution may display the work. The archive inadvertently becomes a free, uncredited content pipeline for proprietary algorithms, risking breaches of trust with the donors, artists, and communities who provided the materials.
Disclosure of Sensitive Information
Collections often contain culturally sensitive indigenous knowledge, raw oral history transcripts, or unredacted personal information intended to be accessed only with proper institutional oversight. Scraper bots do not sign ethics agreements or respect digital embargoes. When they bulk-download these records, they pull sensitive data from their safe archival context and feed it into global databases. Once ingested by an LLM, restricted or harmful information can be permanently baked into the model's generative outputs. This completely bypasses the ethical guardrails the archive spent decades building, turning a curated historical record into an exposed liability.
Know Thy Enemy (Know Thy Bots)
Treating all automated traffic as a single entity is a mistake, and managing bots requires understanding their individual motives and behaviours. What follows is a condensed version of our experiences working with institutions to tell these bots apart, because the right response to a standard search engine differs vastly from that for an evasive data broker.
"If you know the enemy and know yourself, your victory will not stand in doubt". – Sun Tzu's The Art of War
Types of Bots
- Cooperative Actors (Search Engines): Bots like Google or Bing clearly announce their identity and respect your site's speed limits and robots.txt rules. They have a vested interest in keeping your site online and are rarely the source of outages.
- Oblivious Actors (AI Crawlers): These bots scour the web specifically to harvest training data. While major players declare who they are, their default crawling speeds are often far too aggressive for the archival sites’ servers, and are joined by a myriad of undeclared research scrapers that rarely obey site rules.
- Adversarial Actors (Commercial Scrapers): Run by data brokers and aggregators, these bots actively try to evade detection. They constantly change their identifying information, use distributed networks to mimic everyday human browsers, and require active monitoring to block.
Identifying the Machine
Because evasive bots try to look human, defenders must rely on a cluster of behavioural fingerprints rather than just a technical name. Strong signals of automated activity include:
- Speed and Spread: A single user requesting hundreds of pages per minute, or systematically hitting every single URL in a massive digital catalogue in rapid succession.
- Superficial Engagement: Visitors that never execute JavaScript, never load follow-up assets like images, or arrive without the standard session cookies a normal web browser would generate.
- Impossible Geographies: A single session that appears to jump between continents in a matter of seconds.
- Outdated Fingerprints: Bots often masquerade as running on older browser frameworks, creating impossible combinations, such as an outdated version of Internet Explorer running on the newest mobile operating system.
Takeaways
Internet traffic that used to come mostly from people now comes largely from bots, and public archives are getting hit hard because they hold exactly what the AI companies that send the bots out want: clean, well-organised information.
This isn't only a technical problem, it’s also a financial one, straining budgets as well as servers. It also raises real ethical questions. When collections get scraped wholesale, they can lose the context, copyright protections, and cultural safeguards that archivists built around them.
Protecting these collections means giving up on the idea that bots will behave. Being open-access doesn't mean leaving the door unlocked. It means managing traffic more effectively: filtering at the network edge, blocking bots that try to evade detection, caching large files, and enforcing the licensing and privacy terms the collection depends on. We work directly with our clients to manage the impact of bots while upholding their institutional values and preserving public access without compromising the trust, context, and safeguards their collections depend on.
So what does "open access" mean today? It may mean drawing a clear line. An archive's obligation is to the public, to the communities it represents, and to the donors who trusted it with their material, not to bots harvesting its holdings without regard for any of those things. With layered, deliberate defences, institutions can protect their infrastructure and ethical commitments simultaneously, preserving the past while adapting to how the web actually works now.
Do you have a story that you’d like to share? Do you want to learn more about how this relates to your collection? We’re keen to hear from you and answer any burning questions. Get in touch with us through our socials (Facebook, Linkedin and Instagram) or reach out via email.
Tyler