The Real Cost of Unregulated Scrapers
Your website is being scraped, and you are paying for the bandwidth.
For decades, the deal was simple. Search engines crawled your site, indexed your pages, and sent traffic back to your business. It was a symbiotic arrangement. But massive LLM models changed that overnight. Now, training bots harvest your archives to explain your concepts back to searchers on their own platforms. They do not click your newsletter links, they don't view your ads, and they certainly do not pay for your work. They just copy the text.
This shift has turned crawling from a mild server tax into a high-stakes business challenge. If you run a platform or maintain an archive, you face a tough architectural decision. Do you block every AI agent and risk disappearing from the future of discovery? Or do you keep the doors open and watch your intellectual property feed competitors?
To answer this, we need to move past simple emotional reactions and look at the actual data. Not all AI crawlers are the same, and blocking them all is a blunt instrument that can easily backfire. In platforms like Beehiiv, creators are already wrestling with these toggles. We have to separate the bots that drive value from the ones that merely consume resource bandwidth.
Tracking Bots in the Log Files
If you want to know who is accessing your infrastructure, your analytics dashboard won't cut it. You must look at raw server logs.
This is the only source of truth. Every request to your site leaves a footprint containing a user-agent header. Tech giants like OpenAI run multiple crawlers, and they operate differently. For instance, GPTBot is an automated system that crawls the web to train foundation models. It runs in the background. On the other hand, OAI-SearchBot retrieves pages specifically to feed search results and citation interfaces.
Then you have user-triggered fetchers like ChatGPT-User or Perplexity-User. These fire in real-time when a human enters a prompt that requires fresh data from the live web. Here is the catch: these user-initiated fetchers do not always honor your static robots.txt policies. OpenAI openly admits that ChatGPT-User bypasses those files. Because the request is triggered by an active human user, they argue it should be treated as a direct request rather than a background crawl.
If you rely solely on robots.txt, you are missing half the picture.
To get a real handle on this, download a 30-day sample of your logs and count the HTTP requests by user-agent. Look for concentrations. Are these bots hitting your homepage, or are they scraping your high-value product tables? If a scraper spends days downloading product directories, it is not helping your brand. It is cloning your database.
Weighing the Returns Against Crawler Costs
Let's talk about the hard numbers. Bandwidth is cheap, but it isn't free.
At scale, aggressive AI scrapers can disrupt your web server performance. Standard search indexers pace themselves. AI compilers, however, often run threaded scraping runs that mimic denial-of-service attempts. If your hosting environment is billable by usage, a sudden spike from training crawlers will directly increase your end-of-month cloud invoice.
But the bigger issue is traffic cannibalization. When a training bot reads your content and feeds it to an LLM, that LLM can answer queries without ever sending the user to your site. This reduces direct page views, destroying the monetization path for ad-supported business models.
Yet, there is a counter-argument. What happens if you block them all?
You vanish from the index. If traditional search engines continue to morph into generative answering engines, being blocked means your products won't exist in their knowledge base. Your competitors who leave their doors open will get the citations instead. Even if the current traffic volume from LLMs is low, it represents a brand channel that you cannot afford to ignore entirely. It is a classic platform coordinate: how do you balance immediate scrape protection against long-term brand invisibility?
Analytics engines are struggling to help us measure this. Google Analytics recently rolled out an 'AI Assistant' traffic channel, but it is limited. It tracks referrers from ChatGPT, Claude, and Gemini, but it misses platforms like Perplexity that strip referrer headers or use varying redirects. To make a logical decision, you have to measure both citation frequency and referrer traffic before hitting the shutdown switch. You cannot optimize what you do not count.
A Strategic Defense Action Plan
Since you cannot rely on simple text rules, you need a multi-layered approach.
First, realize that robots.txt is not a security tool. It is a public file. As documentation from Google and the robots.txt standards committee makes clear, it is simply a voluntary system for polite web crawlers. It does not hide content, nor does it block malicious actors. Anyone can download your robots.txt, inspect your disallowed paths, and crawl those exact directories anyway.
Instead, use a Web Application Firewall (WAF) to police the perimeter. Modern WAFs from suppliers like Cloudflare or AWS let you create rules that target automation directly at the DNS or proxy level. If a crawler refuses to slow down or masquerades under a fake user-agent, you can drop the connection before it hits your database.
Here is your defense plan:
- Audit your logs: Learn who is visiting and what they are downloading. Focus on the ratio of training bots to search-citation bots.
- Enforce granular blocks: Do not go for a blanket ban. Block training engines (like
GPTBotorcohere-ai) while allowing search systems that provide direct links. - Deploy boundary rules: Set up rate-limiting on your heavy API backends and search pages. If a browser is trying to load 2000 pages per minute, force a challenge.
- Protect core assets: If you have unique databases, reviews, or private pricing lists, put them behind login walls where crawlers cannot touch them.
By organizing your site architecture properly, you protect the crown jewels without making yourself invisible. You can learn more about how to structure layouts on our page about AI-first site architecture.
Ultimately, this is about control. The open web built on unchecked access is gone. The publishers who survive the next wave will be those who actively manage their interfaces, treating crawlers not as inevitable visitors, but as business partners who must pay their way.