The Five-Character Failure in Audits
I ran the new Lighthouse Agentic Browsing category on my site. It was a train wreck. Well, mostly a train wreck. Six audits ran. Three shrugged and said 'not applicable' because headless Chrome couldn't detect the custom execution context APIs. Two passed cleanly.
Then there was the llms.txt check. It failed.
The error message was a joke: 'File does not appear to contain any links.' Now, my file is five kilobytes of pure, curated markdown and plain text. It lists every primary endpoint, the blog feeds, audio player patterns, and full documentation paths. To say it had 'no links' is like saying a phone book has no numbers because they aren't formatted with country codes. But there it was — a big, red fail.
As a cloud application developer, I am used to tools being overly picky. We deal with strict linters and syntax parsers every hour. But when a file is literally extension-named .txt and served with a text/plain MIME type, you expect the parser to show some flexibility. Not here. Lighthouse 13.3.0 introduced this audit, and it doesn't bend. It wants what it wants, even if the result makes no sense on the surface.
The Markdown Parsing Reality Check
Here's the twist. The parser doesn't care that the file is served as text/plain. It doesn't care that the extension is .txt. The team at Chrome decided to build their parser around a strict reading of the llmstxt.org specification.
What does that spec say? It says sections should contain lists of Markdown links.
My plain text links looked like this:
- Homepage: / - Publication masthead and latest articles
To Lighthouse, that is invisible. It represents nothing. The parser is completely blind to plain-text urls or simple text-to-href layouts. It demands the markdown link syntax:
[Homepage](/): Publication masthead and latest articles
That is it. Wrapping the label in square brackets and the URL in parentheses. That is the five-character fix. By replacing the dash separator with a colon and wrapping the targets in standard Markdown markup, the raw text becomes a parseable object. I did that, ran the CLI tool again, and the audit score jumped from 0.67 to a perfect 1.0.
The file is still served as plain text. The extension didn't change. The content of the descriptions remained identical. Yet, in the eyes of Google's machine, the file went from broken to perfect in five minutes. This shows how developer tools prioritize execution over utility.
Mechanical Compliance is Not Content Quality
Let's be clear about what this check is actually measuring. It's checking syntax, not utility.
It is a mechanical parse test. If you auto-generate a completely empty or generic llms.txt file using a standard SEO plugin, you will get a green check. In fact, popular plugins like AIOSEO now generate these files automatically with markdown links by default. They pass without you lifting a finger. But does that mean those files are actually useful to a real AI agent?
Probably not. Most auto-generated files are thin garbage. They contain default pages and lack any deep context about what your application actually does.
Meanwhile, a hand-written page with rich descriptions and deep instructions fails if a developer doesn't style the links exactly right. That is a massive gap. We are optimizing for the parser, not the reader. It is the classic validation trap: prioritizing the shape of the container over the quality of the water inside it. We see this all the time with REST APIs and database schemas, where we validate parameters but fail to check if the database is actually returning correct data. This audit suffers from the exact same limitation.
The Structure Underlying Agent Navigation
Why does this strict formatting matter to the agents? Because parsers are lazy.
If you are building an AI agent to browse the web, you don't feed it the full DOM. The DOM has too much clutter. If you dump a giant page full of nested divs, script tags, and CSS inline declarations into a model's context window, it burns tokens like crazy. It gets confused. That's why models rely on the accessibility tree. We've written before about how the accessibility tree decides what agents read. It's a clean, semantic view of your elements.
Lighthouse's agentic audits are trying to verify if your site provides these clean targets. When an agent searches for /llms.txt, it expects a markdown document that it can parse using regular expressions or library-standard markdown parsers. If your file is raw text, the regex fails. The agent moves on, or worse, hallucinates links that aren't there.
So yes, using the markdown syntax is necessary. It ensures the machine doesn't have to guess. But developers should remember that the accessibility tree and llms.txt are two sides of the same coin: they both exist to present a clean, non-visual interface to automated crawlers. If either one is malformed, your integration breaks.
Don't Build For Bot-Only Environments
There is a growing temptation to build entirely separate websites designed purely for AI bots. Don't fall for it.
John Mueller from Google recently warned about this, urging developers to focus on clean HTML and semantic pages rather than building fork versions of pages. We analyzed this trend showing why you should stop building agent-only websites. When you create parallel pages, the copies drift. One gets updated, the other gets forgotten.
The same problem happens with llms.txt. If you let a plugin auto-generate a raw list of URLs for your site, it will pass the audit, but it won't help the agent complete real tasks. You need to write these files yourself, define what the endpoints do, and format them correctly. Use the audit to fix the formatting, but don't let it become a proxy for actual utility.
I have seen pipelines where people maintain separate files for separate agents, creating an administrative headache. It is much better to have one solid, markdown-formatted llms.txt file that accurately reflects your site's structure, and keep it updated along with your main codebase. That way you pass the Lighthouse test and provide real value to the machines.
Running Your Own Agentic Audits
If you want to run this yourself, the process is simple. You need Chrome DevTools or the Lighthouse CLI.
Run this command in your terminal:
npx lighthouse@latest https://yourdomain.com --only-categories=agentic-browsing
Look at the results. If the llms-txt audit fails, go to your root directory. Open the file. Wrap the URLs. It's ten minutes of manual labor or a quick regex script if you have a massive file. But once it passes, don't close the tab. Look at the actual text you wrote. Does it explain how your API parameters work? Does it tell an LLM how to place an order?
If it doesn't, a green score in Lighthouse is useless. The audit is a starting point, not the finish line.
At the end of the day, building for the agentic web is about creating resilient pathways for software. Formats like llms.txt are a neat shortcut, but only if they are implemented with both physical correctness and actual content depth. Don't just tick the boxes for Google's new audits. Make sure your site runs as well under the hood as it looks on the screen.