What robots.txt Actually Does (and Doesn't Do)
A robots.txt file instructs compliant web crawlers — including Googlebot — about which paths on your site they are permitted to fetch. It is a crawling directive, not an indexing directive. This distinction is critical and widely misunderstood: blocking Googlebot from a URL via robots.txt does not remove that URL from Google's index. If other pages link to a blocked URL, Google can still index it as a link target — it just can't read its content. The only reliable way to prevent indexing is a noindex meta tag or HTTP header on the accessible page.
The One Rule You Must Never Accidentally Deploy
The single most catastrophic robots.txt mistake is deploying Disallow: / under User-agent: *. This tells every compliant crawler to stop indexing your entire site. It is shockingly easy to introduce accidentally — it's the default state of many staging environments, and site migrations frequently push a staging config to production. Validate every deployment with our Robots.txt Validator before it goes live.
Paths You Should Almost Always Block
- /admin/ and /wp-admin/: Backend paths add no SEO value and waste crawl budget.
- /checkout/ and /cart/: Transactional pages behind session state are not indexable anyway and confuse crawlers.
- /?s= (WordPress search results): Search result pages create massive duplicate content at scale. Always block them.
- /tag/ (if duplicate of categories): Tag archive pages on blogs frequently duplicate category page content. Block if they provide no unique value.
Paths You Must Never Block
- /wp-content/uploads/ or equivalent media paths: Blocking these prevents Googlebot from rendering your pages, since it cannot load the images and CSS it needs for visual assessment.
- CSS and JS files: Modern Google rendering requires access to your stylesheets and JavaScript to assess page experience signals. Never block them.
The Sitemap Declaration
Always declare your sitemap location at the bottom of your robots.txt file: Sitemap: https://yourdomain.com/sitemap.xml. This is not required, but it gives crawlers a direct reference to your most important URLs, accelerating discovery of new content and re-indexing of updated pages.