Robots.txt
Robots.txt is a plain-text file at a site's root that tells compliant crawlers which paths they may or may not request, using Allow and Disallow rules per user-agent. It controls crawling, not indexing, and can also point to XML sitemap locations.
Why It Matters
It steers crawlers away from low-value or sensitive areas and preserves crawl capacity for pages that matter. A misconfigured file can accidentally block critical sections, quietly removing large parts of a site from search results.
Common Mistake
Using Disallow to keep a page out of the index. Blocking a URL in robots.txt prevents crawling but the URL can still be indexed without a snippet; use a noindex meta tag or header for reliable removal instead.