Robots.txt
Also known as: Robots Exclusion Protocol, Robots Exclusion Standard
Robots.txt is a plain-text file at a site's root that tells compliant crawlers which paths they may or may not request, using Allow and Disallow rules per user-agent. It controls crawling, not indexing, and can also point to XML sitemap locations.
Key Takeaways
- Robots.txt is a plain-text file at a site's root that tells compliant crawlers which paths they may or may not request.
- It uses Allow and Disallow rules per user-agent and can point to XML sitemap locations.
- Robots.txt controls crawling, not indexing: a disallowed URL can still be indexed without a snippet.
- For reliable removal from search results, use a noindex meta tag or header instead of a Disallow rule.
- A misconfigured file can accidentally block critical sections and quietly drop large parts of a site from search.
How It Works
Robots.txt sits at the root of a domain and is the first file most crawlers request. It lists rules grouped by user-agent, using Disallow to name paths a crawler should not request and Allow to carve out exceptions. Compliant bots honor these rules; the file is a directive to well-behaved crawlers, not a security barrier.
Its main job is steering crawl. By keeping bots out of low-value or duplicative areas, it protects Crawl Budget for the pages that matter, and it can list the location of your XML Sitemap so crawlers find your important URLs faster.
The critical distinction is crawling versus indexing. Blocking a URL in robots.txt stops the crawl but does not guarantee removal: Google can still index the address, often showing it without a description, if other pages link to it. That is why a blocked page can appear in Index Coverage reports as indexed though blocked. To truly keep a page out of results, allow crawling and use a noindex tag, and use a Canonical Tag to consolidate duplicates rather than disallowing them.
Why It Matters
It steers crawlers away from low-value or sensitive areas and preserves crawl capacity for pages that matter. A misconfigured file can accidentally block critical sections, quietly removing large parts of a site from search results.
Example
A SaaS site wants to keep its internal search results pages out of Google. The team adds Disallow rules for the search path in robots.txt, but the URLs still appear in results without snippets because other pages link to them. They fix it by allowing the crawl and adding a noindex meta tag, which reliably removes the pages from the index.
Common Mistake
Using Disallow to keep a page out of the index. Blocking a URL in robots.txt prevents crawling but the URL can still be indexed without a snippet; use a noindex meta tag or header for reliable removal instead.
Frequently Asked Questions
Does robots.txt remove a page from Google?
No. Robots.txt blocks crawling, not indexing. A disallowed URL can still be indexed, usually without a snippet, if other pages link to it. For reliable removal, allow crawling and use a noindex tag or header.
Where does the robots.txt file go?
At the root of the domain, reachable at yoursite.com/robots.txt. Crawlers only look there. A robots file placed in a subdirectory is ignored, and each subdomain needs its own file at its own root.
What is the difference between Disallow and noindex?
Disallow in robots.txt tells crawlers not to request a URL, but the URL can still be indexed. Noindex, set via meta tag or header, tells engines to keep a crawlable page out of the index reliably.