How to Check If ChatGPT, Claude, or Perplexity Are Blocked From Your Site

Plenty of sites are invisible to AI assistants for a reason nobody ever checks: a single line in robots.txt, written months or years ago for an unrelated reason, quietly blocking every AI crawler that would otherwise read the site. Here's how to actually check whether that's happening to you, without guessing.

Step 1: look at the raw file yourself

Go to yourdomain.com/robots.txt directly in a browser. If you get a 404, there's no file at all — which by default means every crawler, AI included, is allowed everywhere, according to Vizup's guide to checking AI crawler access. If the file loads, read it top to bottom looking for User-agent: lines naming any of these:

• GPTBot (OpenAI)
• ClaudeBot, Claude-User, Claude-SearchBot (Anthropic — three separate bots, see below)
• PerplexityBot
• Google-Extended (Gemini training)
• CCBot (Common Crawl, which several AI labs license data from)

A Disallow: / under any of these names blocks that specific crawler from your entire site.

Step 2: understand that Anthropic runs three separate bots

This is the mistake almost everyone makes. Anthropic operates three independent crawlers, each doing a different job — blocking one does nothing to the other two, according to Anagram's breakdown of AI crawlers:

| Bot | Job | Blocking it means | |---|---|---| | ClaudeBot | Collects data for model training | Your content won't be used to train future Claude models | | Claude-SearchBot | Powers search-style indexing | Claude can't index your pages for search-style queries | | Claude-User | Fetches a page live, on a user's direct request | Claude can't read your page even when someone explicitly asks it to |

Most site owners who think they've blocked Claude have only blocked ClaudeBot — Claude-User still works, or vice versa. Check all three names explicitly, not just one.

Step 3: use a dedicated checker, not Search Console

Google Search Console has a robots.txt tester, but it only validates rules against Googlebot and Google-Extended — it has no idea whether GPTBot or PerplexityBot can get in, since Vizup's guide notes Search Console simply doesn't cover non-Google crawlers. Our free robots.txt tester checks all the major AI crawler user-agents against your actual file in one pass, so you don't have to parse the syntax by hand.

Step 4: check for accidental full-site blocks

The most common real-world mistake isn't a deliberate AI block — it's a leftover rule from a staging-site migration or a security scare that disallows / for every user-agent, according to Mo Agency's robots.txt audit guide. If your file has a bare User-agent: * / Disallow: / at the top with no exceptions, every AI crawler is blocked along with everything else — check this first before hunting for AI-specific rules.

Step 5: check server logs, if you have access

The most direct proof isn't robots.txt at all — it's whether a crawler actually requested a page and what status code it got back. If your logs show 403 (forbidden) responses to GPTBot or ClaudeBot requests, something beyond robots.txt (a firewall rule, a bot-blocking service, a CDN setting) is rejecting them even if your robots.txt looks fine, per xSeek's guide to GPTBot. robots.txt is a request, not an enforcement mechanism — a misconfigured WAF can block a "well-behaved" crawler that's honoring your robots.txt perfectly.

What to actually do with what you find

If a crawler is blocked and you want it in: remove the specific Disallow rule for that user-agent, or add an explicit Allow: / line above the wildcard block.

If you're blocking deliberately (e.g., opting out of training data): that's a legitimate choice — just be clear-eyed about the tradeoff. Blocking ClaudeBot for training doesn't block Claude-User, so you can still show up when someone directly asks Claude about you, without contributing to its training set.

Either way, verify after you change it. Re-run the check — don't assume the edit worked. A typo in a user-agent name (Claudebot instead of ClaudeBot) silently does nothing.

For the deeper mechanics of what these crawlers actually do once they're let in, see our investigation into whether AI crawlers really visit your site.

FAQ

How do I check if GPTBot can access my site? Check your robots.txt file for a User-agent: GPTBot block with a Disallow: / rule, or run it through a dedicated checker like our free robots.txt tester that tests against the real GPTBot user-agent.

Does blocking ClaudeBot block all of Anthropic's crawlers? No. Anthropic runs three separate bots — ClaudeBot, Claude-SearchBot, and Claude-User — each with its own name and its own job. Blocking one has no effect on the other two.

Can Google Search Console tell me if AI crawlers are blocked? No. Search Console's robots.txt tester only validates against Googlebot and Google-Extended. It has no visibility into GPTBot, ClaudeBot, or PerplexityBot rules.

My robots.txt looks fine but AI bots still aren't reading my site — why? robots.txt is a request crawlers voluntarily honor, not an access-control mechanism. A firewall, bot-blocking service, or CDN rule can reject a crawler outright regardless of what robots.txt says — check server logs for 403 responses to confirm.