Does ChatGPT Actually Crawl Your Website? What GPTBot, ClaudeBot, and PerplexityBot Really Do

We wanted a simple answer: does anything from OpenAI, Anthropic, or Perplexity actually visit our site? So we did what most people would do — we opened our analytics dashboard and looked for GPTBot, ClaudeBot, or PerplexityBot in the traffic.

We found nothing. Not a single hit, from any of them, ever.

That's not because they weren't crawling us. It's because we were looking in a tool that structurally can't see them — and figuring out why turned into a more useful lesson than the original question.

Why our own analytics couldn't answer the question

Our web analytics, like most product analytics (PostHog, GA4, and similar tools), works by running a small piece of JavaScript in the visitor's browser that reports "someone viewed this page" back to a server. That works great for human visitors, because browsers execute JavaScript.

AI crawlers almost never do. GPTBot, ClaudeBot, and PerplexityBot fetch a page's raw HTML directly, the same way a classic search engine crawler does — they don't load a browser, and they don't run the tracking script that reports the visit. So a bot can read every word of a page in full, and a JavaScript-based analytics tool will show absolutely nothing happened. We weren't missing bot traffic in our dashboard. We were using a tool that was never built to detect it in the first place.

So what do these crawlers actually do?

Once we stopped looking in the wrong place, the real behavior of each bot turned out to be well documented — just not by anyone checking their own analytics.

GPTBot (OpenAI). Introduced in August 2023, GPTBot collects publicly accessible web content that may be used to train future GPT models, and it identifies itself by a specific user-agent string documented directly by OpenAI, along with its published IP ranges, at OpenAI's own GPTBot documentation.

ClaudeBot (Anthropic). Anthropic actually operates three separate crawlers with different jobs, not one: ClaudeBot handles model training data collection, Claude-User fetches pages in real time when a person asks Claude something requiring live web content, and Claude-SearchBot works on search result quality. Anthropic's own support documentation lays out exactly what blocking each one costs you — block ClaudeBot and you're opting out of training data collection; block Claude-User and Claude can no longer fetch your page live when someone asks about it directly, according to Claude's official help center.

PerplexityBot. Runs more like a live search engine crawler — when someone asks Perplexity a question, it can fetch and cite pages close to real time, which is part of why fresh content shows up in Perplexity's answers faster than it tends to influence a model's trained-in knowledge.

All three, and every other legitimate AI crawler, identify themselves through their user-agent string and respect robots.txt — which means the real answer to "are they crawling me" was sitting in our server logs the entire time, not in our analytics tool, according to Contently's explainer on how AI crawlers work.

The actual way to check

If you want to know whether AI crawlers are visiting your site, there are exactly two reliable methods — and neither of them is your normal analytics dashboard:

1. Check your raw server access logs for the user-agent strings GPTBot, ClaudeBot, Claude-User, Claude-SearchBot, and PerplexityBot. This is the ground truth — if your hosting doesn't give you easy log access, ask your host directly; most do.
2. Check whether your robots.txt even allows them in. If a crawler is blocked, it won't visit at all, and no log will ever show it — which means the first fix, before anything else, is confirming you haven't accidentally shut the door. Search Engine Land's coverage of GPTBot's launch noted that many sites blocked GPTBot reflexively without realizing the tradeoff: no training-data crawl also often means no citation in that model's future answers.

Our free robots.txt tester checks exactly this — which AI crawlers your current robots.txt allows or blocks — in under a minute, no server log access required.

What this means if you care about AI visibility

The uncomfortable twist: even confirming a crawler visited your site doesn't confirm you'll be cited. Crawling is necessary but not sufficient — being fetched just means the content was read, not that it was chosen for an answer. That's a separate, harder problem, and it's the one we cover in what AI search visibility actually means and how to measure it.

And if you were hoping a well-crafted llms.txt file would settle all of this — it mostly won't. The crawlers named above overwhelmingly fetch raw HTML directly and ignore it, which we cover honestly in our llms.txt research.

FAQ

Can I see AI crawler visits in Google Analytics or PostHog? No, not reliably. These tools depend on JavaScript execution in a browser, and AI crawlers fetch raw HTML without running a browser or executing scripts. A bot can read your entire page and your analytics will show zero activity.

How do I actually check if GPTBot or ClaudeBot visited my site? Check your raw server access logs for the user-agent strings directly, or ask your hosting provider for log access if you don't have it already. That's the only reliable ground truth.

If I block GPTBot, does that stop me from appearing in ChatGPT? It stops your content from being collected for model training. It doesn't necessarily block live features like ChatGPT's web browsing, which may use a different crawler/user-agent depending on the product surface — check the specific bot names in your robots.txt rules.

Does being crawled guarantee I'll be cited in AI answers? No. Crawling means the content was fetched and readable. Citation is a separate decision the AI system makes based on relevance, structure, and trust signals — being crawlable is necessary but nowhere near sufficient.