Stop Building Agent-Only Websites: Mueller on Why Clean HTML Wins
SEO practitioners love chasing shiny objects. Tell them some new tech trick will boost their ranking, and they will rebuild their entire front-end by Tuesday. The latest craze is creating parallel markdown versions of normal web pages. We are seeing companies spend hundreds of developer hours formatting clean text files for AI agents, while their actual human visitors get stuck navigating a broken layout. It is classic over-optimization, and it is built on a misunderstanding of how crawls actually work.
Google's John Mueller recently put his foot down. His verdict is simple: stop building agent-only mirrors. Just fix your actual website.
Stephanie Walter's Bluesky Post Sparked the Debate
The conversation started when designer Stephanie Walter published a post on Bluesky pointing out a frustrating trend. Developers are bending over backward to offer markdown mirrors and text configuration files (like LLMs.txt) for AI crawlers. Yet, at the exact same time, these very same websites lack basic landmark structures, proper heading hierarchies, and aria labels that human screen readers actually need to browse.
We are literally making websites accessible to silicon while ignoring humans. It is backward.
It highlights a major logic gap. If a site is properly built with standard semantic markup, AI agents can read it without any issues anyway. Standard HTML has worked for crawlers for over thirty years. Rebuilding the document object model just so a bot can read it faster is a solution looking for a problem. You do not need to rewrite your site's templates into pure text.
Mueller's Direct Answer: Just Fix the Core Site
Mueller's reply to Walter's observation was direct and left little room for nuance. He pointed out that a clean, well-made site works well for everyone: AI agents, search engines, LLMs, and actual human users. He noted that trying to solve accessibility or content delivery issues by building separate agent-friendly versions is just generating technical debt. You will end up redoing it again and again.
"Just fix it," he said. That is the ultimate takeaway.
Think about the long-term maintenance of this. Every time your marketing team updates a product page, your system has to sync it to a markdown alternative. If the sync fails, you are presenting outdated information to LLMs and fresh information to humans, or vice versa. It is a maintenance disaster waiting to happen. You do not want to double your workload for zero benefit.
The Fallacy of Building Separate Mirror Pages
Why is there such a rush to build these text files? Part of it is fear of missing out. The search optimization industry is notoriously reactionary. If a prominent competitor adds a file to their root directory, the rest of the industry rushes to copy it without checking if it actually drives value.
It is very similar to early link-building tactics. Years ago, a directory owner could buy a few high-PageRank links, push his homepage to a PageRank of 7, and sell directory listings for thousands of dollars. SEOs lined up to pay him. No one stopped to measure if those links actually improved rank. They did not, but the hype was too loud for logic.
The markdown craze feels identical. We are wasting engineering time on mirror pages because it feels digital-native, not because anyone has proved it helps.
Clarifying OAI-SearchBot and the Cloudflare Trend
The trend gained massive steam when Cloudflare introduced automated markdown generation. The company marketed it as a way for websites to save on LLM token costs when crawlers scraped their content. In their announcement, Cloudflare claimed that OpenAI’s crawler, OAI-SearchBot, was actively consuming these markdown alternatives.
Except OpenAI's official docs say something completely different.
According to OpenAI's documentation, OAI-SearchBot is designed to crawl standard websites to surface them in ChatGPT search results. The documentation advises webmasters to allow the bot in robots.txt and whittle down IP blocks, but it does not mention or recommend using markdown. The bot is looking for standard HTML. It can read markdown if you force-feed it, but it does not prefer it. The entire premise of the Cloudflare push is built on a feature that crawlers do not even require. You can read more about the technical details of markdown-for-AI trends in Search Engine Journal's breakdown of the markdown trend.
Agentic Browsers are Coming: Will Quality Principles Change?
The debate extends beyond simple markdown files. Search marketer Jason Kilgore asked Mueller another critical question on Bluesky. With AI safety and agent capabilities advancing, tools like Gemini 3.5 Flash can now browse the web autonomously using "Computer Use" features. If an AI agent tools around your site on behalf of a human user, do Google's search quality principles change?
For example, does the quality guideline that "images must provide a satisfying experience" evolve when the user is an AI agent rather than a human?
Mueller expects that the core principles will remain steady. A site that provides a satisfying experience for a human user will naturally be useful for an agentic browser. Modern engines rely heavily on user interaction and popularity signals. The underlying goal remains the same: cater to the end user. You can find the context on how Gemini's browser access interacts with these guidelines in the Search Engine Journal report on agentic SEO.
Drawing Parallels to the Nofollow Mistakes of the Past
There are technical details that will change as agents become more active. Mueller specifically warned against one major pitfall: blindly blocking agentic browsers.
We have seen this movie before. When Google introduced the nofollow attribute, site owners went crazy. They set up "PageRank sculpting" schemes, blocking access to basic utility pages like "About Us" and "Contact" in a bid to funnel link juice to product listings. It did not work, and it ended up messing up their site architectures.
If webmasters start blocking AI agent user-agents out of fear of scraping, they might accidentally cut themselves off from critical search features. An agent that cannot browse your site cannot recommend your product to the user who sent it.
Practical Next Steps for Sustainable Technical Layouts
So what should you actually do instead of building markdown mirrors?
Start by auditing your header structures. Ensure you have one clear H1 per page, followed by logical H2 and H3 structures. Use standard HTML landmarks like <main>, <nav>, and <footer> so that both human screen readers and AI agents know exactly where the content lives. Keep your internal links clean and crawlable.
If you focus on real accessibility for human beings, you automatically solve the crawlability equation for AI. Do not build technical debt. Just build a better website.
