Robots.txt Validator
Free online robots.txt validator. Test and validate your robots.txt file — check if URLs are allowed or disallowed for Googlebot and other crawlers. Prevent SEO disasters before they happen.
Enter robots.txt rules and a URL to test above.
How to Use Robots.txt Validator
Enter Robots.txt Content or URL
Paste your robots.txt content directly, or enter a website URL to fetch its existing robots.txt file. The tool works with both.
Test Against URLs
Enter specific URLs to test against the robots.txt rules. The tool checks whether each URL is allowed or disallowed for a given user agent (default: Googlebot).
Review Results
Each URL gets a clear Allow or Disallow result with the matching rule shown. Fix any URLs that are incorrectly blocked or allowed before deploying to production.
Example Robots.txt
User-agent: *
Disallow: /admin/
Disallow: /wp-admin/
Allow: /admin/login/
Disallow: /search
Allow: /search?
User-agent: Googlebot
Disallow: /private/
Sitemap: https://example.com/sitemap.xmlThis allows all bots except on /admin/ and /wp-admin/. Googlebot is additionally blocked from /private/. The sitemap URL helps crawlers discover all indexable pages.
Frequently Asked Questions
What is robots.txt?
Robots.txt is a text file placed at the root of a website (e.g., example.com/robots.txt) that tells search engine crawlers which pages they can and cannot access. It uses the Robots Exclusion Protocol — a standard that all major search engines (Google, Bing, Yahoo) respect. While it's not a security mechanism (malicious bots can ignore it), legitimate crawlers follow robots.txt rules strictly.
What happens if my robots.txt is wrong?
A misconfigured robots.txt can have serious consequences. Accidentally disallowing your entire site (Disallow: /) tells search engines to index nothing — your pages will disappear from search results. Accidentally allowing admin pages or staging servers exposes them to search indexing. Always validate your robots.txt before deploying, especially after site migrations or major updates.
How do I test a specific URL against robots.txt?
Enter the robots.txt rules and the URL you want to test. Our tool checks which rule matches that URL path for the selected user agent. The longest matching rule wins — if you have "Disallow: /admin/" and "Allow: /admin/login/", the Allow rule takes precedence for /admin/login/ because it's more specific.
What are common robots.txt mistakes?
Common errors include: using absolute URLs instead of relative paths (Disallow: https://example.com/page is wrong — use Disallow: /page), blocking CSS and JS files (Google needs to render them for mobile-friendly testing), using Disallow without a user-agent line (robots.txt starts with User-agent:), and forgetting that Allow directives override Disallow for more specific paths.
Robots.txt Best Practices
Robots.txt is a simple file with outsized impact. A single misplaced character can de-index your entire site from Google. Every developer who manages a website should know how to write and validate robots.txt correctly.
Key Rules
- Always start with User-agent: line before any Disallow/Allow directives
- Use relative paths — Disallow: /private/ not Disallow: https://example.com/private/
- Don't block CSS, JS, or image files — Google needs them for rendering
- Include your Sitemap URL at the bottom of robots.txt
- Test every rule change before deploying to production
- Use Google Search Console's robots.txt Tester for additional validation