Robots.txt Generator
Build a custom robots.txt file — block crawlers, protect paths, add your sitemap.
Block crawlers
Search Engines
AI Crawlers
SEO Tools
Disallow paths (for all crawlers)
Note: Googlebot ignores Crawl-delay — use Google Search Console instead.
robots.txt output
Your robots.txt will appear here…FAQs
What is a robots.txt file?
A text file at the root of your website (e.g. yoursite.com/robots.txt) that tells crawlers which pages or sections they can and cannot access.
Does blocking a bot in robots.txt hide my pages?
No — it prevents crawlers from reading the page content, but the URL itself can still appear in search results if other sites link to it. Use noindex meta tags for full exclusion.
Should I block AI crawlers?
It depends on your goals. Blocking AI training crawlers (GPTBot, CCBot) prevents your content from being used in AI training data. Blocking AI search crawlers (PerplexityBot, ClaudeBot) removes you from AI-powered search results.
What is the Disallow: / rule?
Disallow: / blocks the crawler from accessing your entire site. Use it carefully — for example, to block SEO tools from crawling you, but never for main search engines.