Robots.txt Analyzer
Analyze your robots.txt to ensure AI platforms like ChatGPT can discover your content.
Paste your robots.txt content and click "Analyze" to see results
Why Check Your Robots.txt for AI Crawlers?
Many websites inadvertently block AI crawlers in their robots.txt file, preventing their content from being discovered by ChatGPT, Claude, and other AI platforms. This analyzer checks whether major AI crawlers are allowed or blocked on your site.
Important AI Crawlers to Allow
- GPTBot - OpenAI's crawler used to train and update ChatGPT
- ChatGPT-User - ChatGPT's browsing feature when users search the web
- ClaudeBot - Anthropic's crawler for Claude AI
- PerplexityBot - Perplexity AI's crawler for their AI search engine
Best Practices for AI-Friendly robots.txt
- Allow GPTBot and ChatGPT-User for ChatGPT visibility
- Allow ClaudeBot for Claude AI discovery
- Include a Sitemap directive to help crawlers find your content
- Avoid blanket Disallow rules that might block AI crawlers