What is robots.txt?
A file that tells search engine crawlers which pages they are allowed to visit.
Written by DidYouSEO Team·Updated April 2026
Definition
A robots.txt file is a plain text file placed at the root of a website (e.g. example.com/robots.txt) that uses the Robots Exclusion Standard to instruct search engine crawlers which pages or sections they should or should not access. Common directives include 'Disallow' (block a path) and 'Allow' (explicitly permit a path). It can also include a 'Sitemap' directive pointing crawlers to your XML sitemap.
Why it matters for SEO
A misconfigured robots.txt is one of the most damaging technical SEO errors possible. A single Disallow: / line blocks the entire site from being indexed. Blocking CSS or JavaScript files can prevent Google from rendering your pages properly. Conversely, not blocking /admin, /checkout, or other low-value pages wastes crawl budget. Every site should have a valid robots.txt that accurately reflects what should and should not be indexed.
How to check robots.txt
DidYouSEO's robots.txt tester fetches and parses your robots.txt file, checks for common errors (blocking the whole site, blocking CSS/JS, missing sitemap reference), and tells you which pages are blocked for Googlebot.
Check it on your site - free
No credit card. No sign-up required for free tools.