Your robots.txt file is a set of instructions to web crawlers: which pages they can fetch and which they should skip. In the SEO era, most robots.txt configurations were written with Google and Bing in mind. Then a wave of AI crawlers arrived with their own user agents - and many sites are blocking them without knowing it.
A blocked AI crawler cannot index your content. If GPTBot cannot read your pages, ChatGPT cannot cite them. It is as simple and as expensive as that.
The AI crawler user agents you need to know
Each AI company sends crawlers with specific user agent strings. Here are the ones that matter most for AEO:
- GPTBot - OpenAI's crawler for ChatGPT and the OpenAI API retrieval pool
- OAI-SearchBot - OpenAI's real-time search crawler (separate from GPTBot)
- ClaudeBot - Anthropic's crawler for Claude's knowledge base
- anthropic-ai - Anthropic's secondary crawler identifier
- PerplexityBot - Perplexity AI's retrieval crawler
- Googlebot-Extended - Google's crawler for Gemini and Bard content
- CCBot - Common Crawl, used as a training data source by many AI companies
- Meta-ExternalAgent - Meta's crawler for Meta AI (Llama-based products)
- Applebot-Extended - Apple's crawler for Apple Intelligence
The most common blocking mistakes
Mistake 1: Wildcard User-agent with Disallow
The most common mistake is a catch-all rule that blocks all non-Googlebot crawlers. A configuration like this blocks every AI crawler:
This pattern was common advice in the early 2010s to prevent scrapers. Today it also blocks every AI crawler not explicitly named. The fix is to either add each AI user agent explicitly, or restructure to a default-allow pattern.
Mistake 2: Blocking /api/ paths that render content
Many Next.js and headless sites serve their readable content through API routes. If Disallow: /api/ covers paths that a crawler would need to access for core content, you are effectively invisible.
Mistake 3: A malformed robots.txt
A syntax error - missing colon, wrong casing, or a blank line in the wrong place - can cause some crawlers to interpret the whole file as blocking. Validate with Google Search Console's robots.txt tester after any change.
The recommended robots.txt for AEO
This configuration allows all crawlers by default and only restricts paths that should not be indexed. It is the safest pattern for AEO because it does not require you to enumerate every AI user agent as new ones emerge.
What about llms.txt?
llms.txt is an emerging convention (inspired by robots.txt) where you place a file at /llms.txt that gives AI models a structured summary of your site - what it does, key pages, what you want models to know about you. It is not yet adopted by any major AI crawler as a standard, but early adoption is low-cost and potentially high-reward as the standard evolves. CiteRank checks for its presence as part of the AI crawler access audit. A step beyond a static summary is an MCP server, which lets an agent query your data directly instead of just reading about it.
An allowed robots.txt entry only means a crawler is permitted to fetch your pages, not that it actually did. Server log analysis is the only way to confirm GPTBot, ClaudeBot, and PerplexityBot are really visiting - and it catches the WAF rules and JavaScript rendering gaps that block them even when robots.txt looks correct.
Run a free CiteRank audit to check your robots.txt configuration in seconds. The crawler access check flags exactly which user agents are blocked and shows you the specific rules causing the block.