Robots.txt Tester

What is robots.txt?

robots.txt is a file at the root of your domain that tells search engine crawlers which pages they can and cannot access.

Does blocking a page in robots.txt remove it from Google?

No - blocking crawling only prevents Google from seeing the page content. If other sites link to it, Google may still index the URL. Use noindex meta tags to remove pages from search results.

What happens if I don't have a robots.txt?

Crawlers will crawl everything by default. While not having one isn't harmful, having a well-configured robots.txt helps guide crawlers efficiently.

How do I tell Google where my sitemap is?

Add a Sitemap: line to your robots.txt file pointing to your sitemap URL. Example: Sitemap: https://example.com/sitemap.xml