Fast validator with explainable results
Robots.txt Checker& Validator
Validate and test your robots.txt against Googlebot, Bingbot and other crawlers. Catch syntax errors, directive conflicts and precedence issues in seconds.
Enter a website URL to check robots.txt
What the Robots.txt Checker Does
This tool parses your robots.txt, applies the Robots Exclusion Protocol rules, and evaluates crawler access for specific user-agents and URLs.
How It Works
- 1Enter a site URL or paste robots.txt content
- 2Select a user-agent and optional test URL
- 3Get a verdict with matching rule and precedence explanation
What We Check
- Core syntax: user-agent, allow, disallow, comments
- Pattern matching: wildcards and anchors
- Directive precedence: longest match wins
- File placement and accessibility
- Sitemap and vendor-specific directives
Best Practices
- Use robots.txt to manage crawling load, not indexing
- Test specific URLs against intended user-agents
- Prefer minimal, explicit rules over complex patterns
- Keep robots.txt lean and stable for better caching
- Expose XML sitemaps via robots.txt directives
Common Pitfalls
- Don't block CSS/JS files needed for rendering
- robots.txt controls crawling, not indexing
- Changes take ~24h to apply due to caching
- Keep file size well under 500 KB limit
- Test before deploying to avoid blocking important pages
Frequently Asked Questions
Does robots.txt block indexing?
No. Robots.txt controls crawling, not indexing. Use noindex meta tags or authentication to prevent indexing.
How long do changes take to apply?
Google generally refreshes about every 24 hours, so expect a delay before changes are respected.
Is crawl-delay supported by Google?
Google ignores crawl-delay. Bing and some engines may consider it. Use webmaster tools for rate controls.
What is the maximum robots.txt size?
Keep it under 500 KB; content beyond that may be ignored by Google.