Robots.txt Guide:
Rules, Syntax & Validation

Robots.txt: Five Lines That Control Search Engines

Every major search engine — Google, Bing, Yahoo, Yandex, Baidu, DuckDuckGo — looks for /robots.txt before crawling a site. This text file, placed at the root of your domain, tells crawlers which parts of your site they can access and which they should skip. It uses the Robots Exclusion Protocol, a standard from 1994 that remains universally respected by legitimate crawlers. A Disallow: / directive tells every crawler to index nothing. A Disallow: /admin/ directive tells them to skip your admin panel but index everything else. The file is tiny — often 5-10 lines — but its impact is enormous. One wrong directive can de-index your entire site from Google.

A robots.txt validator tests your rules against specific URLs. Paste your robots.txt content and a test URL. The validator shows whether that URL is allowed or disallowed, and which rule made the decision. This catches configuration errors before they affect your search rankings. Always validate robots.txt after making changes — especially after a site migration where old rules might be carried over blindly.

Robots.txt Syntax: The Rules That Matter

User-agent: * — Applies the following rules to all crawlers. You can target specific crawlers: User-agent: Googlebot applies rules only to Google. Rules for a specific user-agent override general rules for that bot. A common pattern: general rules for all bots, then specific rules for Googlebot-Image to control image indexing separately.

Disallow: /path — Blocks the specified path and everything under it. Disallow: /admin/ blocks /admin/, /admin/login, /admin/dashboard/users. The trailing slash matters: Disallow: /admin would also block /administration and /administer because it matches as a prefix. Always use trailing slashes when blocking directories to avoid accidentally blocking paths with similar prefixes.

Allow: /path — Overrides a Disallow for a more specific path. If you have Disallow: /private/ but want to allow /private/public-docs/, add Allow: /private/public-docs/ after the Disallow. The most specific matching rule wins, not the first one. This is the opposite of how most programming logic works and is a common source of errors.

Sitemap: https://example.com/sitemap.xml — Tells crawlers where your XML sitemap is. This line can appear anywhere in robots.txt and applies to all crawlers regardless of User-agent sections. Including your sitemap URL in robots.txt is a best practice — it's the first place crawlers look for discovery hints.

Crawl-delay: 10 — Requests that crawlers wait N seconds between requests. Not all crawlers respect this (Google ignores it; Bing and Yandex do). Use Google Search Console to control crawl rate for Googlebot. Use Crawl-delay for other crawlers that might be too aggressive on smaller servers.

Common Robots.txt Mistakes

Using absolute URLs. Disallow: https://example.com/admin/ is wrong. Rules must use relative paths: Disallow: /admin/. Absolute URLs in robots.txt are silently ignored, meaning the rule you thought was protecting your admin panel does nothing. This is one of the most common mistakes and the hardest to catch without a validator because the file looks correct at a glance.

Blocking CSS and JS. Disallow: /wp-content/ blocks your themes, stylesheets, and JavaScript. Google needs to render your pages to assess mobile-friendliness and visual layout. If you block CSS and JS, Googlebot sees unstyled, broken pages — and your rankings suffer. Only block paths that contain private data or pages you explicitly don't want indexed. Let Google access everything needed to render your pages correctly.

Forgetting the leading slash. Disallow: admin/ without the leading slash doesn't match /admin/. The path must start with /. This is a syntax error that produces no warning — the rule simply doesn't apply. Validators catch this. Manual code review often misses it.

Multiple sitemap directives. You can have multiple Sitemap: lines, one per sitemap URL. But don't list sitemaps that don't exist — crawlers will try to fetch them and log 404 errors. Use our Sitemap Validator to confirm each sitemap URL returns valid XML before listing it in robots.txt.

Testing Robots.txt Before Deployment

The robots.txt testing workflow: write your rules, test with a validator against the 10 most important URLs on your site, then deploy. After deployment, use Google Search Console's robots.txt Tester (free, integrated with your Search Console account) for final verification. Our robots.txt validator gives you instant, offline testing during development — paste your rules, paste a URL, see the result. This is faster than deploying, waiting for Google to recrawl, then finding out your rules are wrong.