Free tool · robots.txt
Is your robots.txt blocking AI crawlers?
Many sites block AI crawlers by accident — a catch-all Disallow, a copied robots.txt, or a CDN setting. If GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot are blocked, ChatGPT and Perplexity cannot read your pages and will not cite them. This checker parses your robots.txt and shows exactly which AI agents are allowed.
We fetch four public files from your site and nothing else. Reports get a shareable link.
What this tool checks
- GPTBot and OAI-SearchBot (OpenAI training index and ChatGPT search)
- ChatGPT-User (live browsing inside ChatGPT)
- ClaudeBot and Claude-User (Anthropic)
- PerplexityBot (Perplexity search index)
- Google-Extended (Gemini grounding)
- Wildcard rules and longest-match precedence, the way Google's robots spec defines them
Frequently asked questions
Should I allow AI crawlers?
If you want to be recommended by AI assistants, yes — at least the search-oriented agents (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User). You can still block training-only crawlers like GPTBot if you object to model training; the two are separate user agents.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects data for training OpenAI models. OAI-SearchBot builds the index that ChatGPT search uses to answer questions with citations. Blocking GPTBot does not remove you from ChatGPT search; blocking OAI-SearchBot does.
My robots.txt allows everything — why does the check still fail?
Some CDNs and bot-protection services (Cloudflare, Vercel firewall, Akamai) block AI user agents at the network layer regardless of robots.txt. This checker reads robots.txt only; if it passes but AI tools still cannot fetch you, look at your WAF settings.
Run it on your site
About 15 seconds. Free, no signup.