The Layer Cake: Six Things Happening, Not One
There are at least six distinct layers building up under the "AI search" label. They’re not competing. They’re stacking.
Layer 1: Crawlable HTML Pages
Still the foundation. Still everything.
No one’s replacing this. Not now, not ever. If your site doesn’t render clean, semantic HTML with real links and structure, none of the rest matters. Crawlers still need to see your nav. Your footer. Your breadcrumbs. Your headings. That’s how they understand your site’s shape.
Layer 2: Schema.org
The original machine-readable hint system. A vocabulary. You tell search engines: "This is a product. This is a review. This is a FAQ."
It had its moment. Rich results. Then gaming. Then Google learned to infer meaning without it. Now it’s a verification layer — not a ranking lever.
Layer 3: LLMs.txt
A navigation file for agents already on your site.
It doesn’t help them find you. It helps them navigate once they’re there.
John Mueller put it perfectly: "If someone is already on your website... maybe some kind of automated system is helpful."
Think of it like a hotel’s internal directory. Useful if you’re already in the lobby. Useless if you’re still outside.
Layer 4: MCP and WebMCP
MCP lets agents talk to backend services — payment systems, inventory APIs, booking engines.
WebMCP lets them interact with live web pages — filling forms, clicking buttons, submitting data.
These aren’t SEO tools. They’re automation protocols. If you’re building a SaaS product or an ecommerce flow that agents might use, this is your future. If you’re a content marketer? Ignore it for now.
Layer 5: OKF — Open Knowledge Format
A directory of markdown files. With YAML frontmatter. Title. Description. Tags. Links.
That’s it.
Google’s own words: "Just markdown. Just files. Just YAML frontmatter."
It’s not a knowledge graph. No typed relations. No explicit semantics. Just documents linked together, and an LLM left to infer meaning.
And here’s the kicker: it was built for internal engineering teams.
Not marketing sites.
Not product pages.
Runbooks. API specs. Metric definitions.
If you’re a dev team documenting a complex system, OKF is brilliant. It’s clean, readable, and easy to version.
If you’re an SEO trying to convert your homepage into markdown to "optimize for AI"? You’re building a parallel version of your site. And you’re stripping away all the semantic structure that search engines rely on.
John Mueller again: "Having a website that uses normal HTML is critical."
Don’t remove your navigation to "save tokens." You’re not saving tokens. You’re breaking discovery.
Layer 6: ARD — Agentic Resource Discovery
Now this is different.
ARD isn’t about your content.
It’s about your capabilities.
Problem: Agents need to be pre-wired to every tool they use. That doesn’t scale.
Solution: ARD.
An ai-catalog.json file on your domain. Lists your tools. Your APIs. Your agents.
A registry indexes those catalogs. Agents find them at runtime. They verify ownership. They connect.
It’s not about what you say. It’s about what you can do.
Hugging Face already adopted it. Others are following.
But here’s the catch: the identity layer underneath is still being settled.
Treat ARD like infrastructure you watch — not build on.
It’s early. The standards aren’t locked. The trust models are still in flux.
And yes — there’s a gap. No agreed media type for OKF bundles yet. Catalogs can list them, but can’t reliably identify them.
That’s not a bug. It’s how specs grow.
Application/json wasn’t officially registered until 2006. JSON was already everywhere by then.
But OKF isn’t JSON. It’s an artifact an agent might act on. Get the type wrong? You risk a system acting on something it shouldn’t trust.
This isn’t a footnote. It’s a red flag.
— A.O.
The Real Shift: Second Layer, Not Third Head
The web isn’t splitting into a third head.
It’s growing a second layer.
A parallel machine-readable infrastructure.
Sitemap.xml told crawlers which URLs existed.
Robots.txt told them where not to go.
LLMs.txt tells agents how to navigate once they’re in.
OKF packages internal knowledge.
ARD advertises capabilities.
All of it is infrastructure.
None of it replaces the human web.
Your site still needs to be findable. Credible. Useful.
Your content still needs to answer questions. Solve problems. Build trust.
The SEOs who understand this layering — who see OKF as a dev tool, ARD as an API spec, LLMs.txt as a hallway sign — they’re the ones who’ll make better decisions.
The rest? They’ll be chasing the next shiny thing.
And the web will keep growing.
— A.O.
