Google Lighthouse - the engine behind PageSpeed Insights - has a new category called Agentic Browsing. It audits how well your site works for AI agents rather than for human visitors, and the change most site owners will notice first is that Lighthouse now checks your llms.txt file and flags it when it does not follow the recommended format. If you have run a recent PageSpeed Insights report and seen "llms.txt does not follow recommendations", this is why.
This guide explains exactly what the Agentic Browsing category checks, the three rules the llms.txt audit applies, why "File does not appear to contain any links" is by far the most common failure, and how to write a file that passes. It also covers an awkward tension worth understanding: Google's own Search team has repeatedly said you do not need llms.txt - yet Lighthouse now audits it anyway.
What the Agentic Browsing category is
Agentic Browsing is a set of deterministic checks that evaluate how machine-readable your site is for AI agents - the software that navigates the web on a user's behalf. Unlike Performance or SEO, it does not produce a weighted 0-to-100 score. Chrome describes the standards for the agentic web as still emerging, so the category currently reports pass/fail signals rather than a single ranking. PageSpeed Insights shows it as a simple fraction, such as 2/3.
The category groups several signals that share one assumption: an AI agent reads a page differently from a person. It relies on the accessibility tree for structure, on stable layout to locate elements, and on machine-readable summaries to understand a site quickly.
- Accessibility tree: every interactive element needs a programmatic name and label, because the a11y tree is the agent's primary model of the page
- Layout stability: Cumulative Layout Shift (CLS) matters because agents act on element positions that must not move underneath them
- llms.txt: a machine-readable summary of your site at the domain root, checked against the llms.txt specification
- WebMCP: an emerging API that lets a site expose its forms and actions to agents directly, audited only under the Chrome WebMCP origin trial
Agentic Browsing is experimental. Testing the full category requires Chrome 150 or later, and the WebMCP audits require registering for the WebMCP origin trial. Chrome notes the category is subject to change - treat its signals as early direction, not a fixed standard.
The three rules the llms.txt audit checks
The llms.txt audit is the part of Agentic Browsing that nearly every site can act on today. It does not grade the quality of your summary or the relevance of your links. It applies three mechanical checks, and failing any one of them produces the "llms.txt does not follow recommendations" warning.
1. It must have an H1 header
The file must begin with a single top-level Markdown heading naming the site or project - for example, # Acme Analytics. In the llms.txt specification this is the only strictly required element. Miss it and Lighthouse reports that the file is missing a required H1 header.
2. It must contain Markdown links
The file must include at least one Markdown-formatted link in the form [Page name](https://example.com/page). This is the rule sites fail most often, and it is worth understanding precisely - which the next section covers. The error text is unambiguous: "File does not appear to contain any links."
3. It must not be suspiciously short
The file has to carry real content - more than roughly 50 characters, according to DebugBear's analysis of the audit. A stub containing only a heading is flagged as suspiciously short. In practice, any genuine llms.txt with a summary and a handful of links clears this easily.
Why "File does not appear to contain any links" is the most common error
The links rule trips up well-intentioned files because the most natural way to list pages is also the wrong way. Many site owners write their llms.txt as a plain bulleted list of paths with descriptions. That reads perfectly to a human, but it contains zero Markdown links, so Lighthouse fails it.
The llms.txt specification requires each entry in a section to be a Markdown hyperlink - [name](url) - optionally followed by a colon and a note. Converting the same list to proper links is all it takes to pass:
Use absolute URLs (https://example.com/blog), not bare paths. The file is meant to be fetched and followed by AI systems, and absolute links remove any ambiguity about where each resource lives.
Pass, fail, or not applicable
Lighthouse resolves the llms.txt audit to one of three states, and the distinction matters when you read a report.
- Pass: the server returns the file successfully and it follows the recommendations above
- Fail: the file exists but breaks a rule, or the server throws an error while retrieving it
- Not applicable: there is no file at all (a 404), in which case Lighthouse marks the audit N/A rather than failing it
That last point is the one people miss. Not having an llms.txt is not scored as a failure - the audit simply does not apply. A broken llms.txt, by contrast, is an active fail. If you are going to publish one, publishing it correctly matters more than publishing it at all.
But Google said you do not need llms.txt
There is a genuine contradiction here worth naming. Members of Google's Search team have publicly downplayed llms.txt, noting that Googlebot and Google's AI systems do not rely on it for crawling or ranking. Yet Lighthouse - also a Google project - now audits it. So which is it?
Both can be true. The Agentic Browsing category is forward-looking tooling for the agentic web, not a statement about Google Search rankings. The llms.txt standard, created by Jeremy Howard at Answer.AI in 2024, is adopted unevenly: some AI tools and documentation platforms consume it, many do not. The honest position is that llms.txt is low-cost insurance rather than a guaranteed ranking factor.
What changed with this release is the cost-benefit balance. The file takes minutes to write, Google now surfaces a visible warning when it is malformed, and a correct file can only help with the agents that do read it. That is a reasonable case for getting it right - and very little case for leaving a broken one in place.
How to write a compliant llms.txt
A file that passes the audit and follows the specification has a predictable shape: an H1, a one-line summary, optional context, then one or more H2 sections of Markdown links. Here is a template you can adapt.
- Start with a single # H1 naming your site - this is the only required line
- Add a > blockquote with a one-sentence summary of what you do
- Write a short paragraph of context if it helps an AI understand the rest
- Group your most important URLs under ## headings as Markdown links, each with a brief note
- Use an "## Optional" section for secondary resources an AI can skip under tight context limits
- Save the file as UTF-8 and serve it at https://yourdomain.com/llms.txt
Do not block /llms.txt in robots.txt, and make sure it returns a 200 with a text/plain or text/markdown content type. A file that 404s, redirects, or returns a server error will either be skipped or actively fail the audit.
WebMCP and where Agentic Browsing is heading
llms.txt is the accessible first step, but the more ambitious part of Agentic Browsing is WebMCP. It is an emerging browser API that lets a site declare its actions - submitting a form, searching a catalogue, booking a slot - so an agent can call them directly instead of guessing from the rendered page. Lighthouse audits WebMCP integrations, but only for sites enrolled in Chrome's WebMCP origin trial.
The two checks that need no opt-in are rooted in fundamentals you should already care about: a well-formed accessibility tree and a stable layout. Semantic HTML with proper ARIA labelling is, in Chrome's framing, the machine-eye view of your page - the same markup that helps screen-reader users helps AI agents parse and act on your content. Investing there pays off regardless of how the agentic standards evolve.
How to check your own site
Run your URL through PageSpeed Insights and look for the Agentic Browsing category, or open Lighthouse in Chrome 150 or later. The llms.txt audit appears under Agent Accessibility, with the exact error text if your file falls short. Fix the file, redeploy, and re-run the report to confirm the warning clears.
CiteRank's free AEO audit now validates llms.txt against these same three rules - H1, Markdown links, and length - alongside the other signals that determine whether AI engines cite you. If you would rather see every issue in one pass than run several separate tools, paste your URL and check.