Bot Access Rules: Which AI Crawlers Your robots.txt Lets In
How robots.txt decides which AI bots can reach your content, and why llms.txt has nothing to do with it.
What it measures
This check reads your robots.txt and works out which AI crawlers can reach your content. It evaluates the file the way a crawler does, following RFC 9309: per-agent groups, Allow overrides, longest-prefix matching, and wildcards. It then reports which named AI bots, such as GPTBot, ClaudeBot and PerplexityBot, are blocked from your site root.
Why it matters for AI
robots.txt is how you tell crawlers what they may fetch. Block an AI agent, whether by naming it or with a blanket "Disallow: /" under "User-agent: *", and it skips your content. Nothing you do to the page itself changes that.
| Status | Meaning | Impact |
|---|---|---|
| All agents allowed | No rule in robots.txt blocks a known AI crawler | AI crawlers can reach your content |
| Some agents blocked | Specific AI bots are disallowed by name | Those bots skip your site |
| All blocked | A blanket Disallow: / applies to every agent | No AI crawler reads your site |
robots.txt and llms.txt do different jobs
These two get conflated constantly, so to be plain about it: llms.txt has no blocking semantics. It cannot allow or deny a crawler, and no version of the spec has suggested it could. It is a curated index of what is worth reading, used once an agent already has access.
Access is robots.txt. Curation is llms.txt. We grade llms.txt as its own metric.
How to improve
- Read your robots.txt - visit yoursite.com/robots.txt and look at every Disallow rule
- Remove a blanket Disallow: / - under User-agent: * it blocks everything, AI crawlers included
- Decide about named AI bots deliberately - many sites inherit these rules from a plugin default without ever choosing them
- Block paths, not domains - disallow the directories you want kept out rather than the whole site
💡 Quick win
Open yoursite.com/robots.txt now and search for "Disallow: /" under "User-agent: *". That single line, if it is there, is why AI assistants never quote you.