AI Crawler User Agents: The 2026 Reference List for Your robots.txt and Logs

Every robots.txt decision, log investigation, and "who's reading my site?" question starts the same way: knowing the AI crawler user agents by name. This is the reference list - the major AI bots as of 2026, what each one actually does, and the exact token you'd use to allow or block it.

Bookmark it; the names matter more than they used to, because bots are now most web traffic and nearly half of AI bot activity is training collection.

The AI crawler user agent list (2026)

Primary sources for the big four: OpenAI's bot documentation, Anthropic's crawler documentation, Perplexity's crawler docs, and Apple's Applebot page. Operators change behavior and add bots regularly - always check the vendor page before writing rules you'll forget about.

Reading the categories

The category column is the part that should drive your policy - the same bot-intent framework we use in DidYouSEO's bot traffic tracker:

Training crawlers collect pages to train future models. Blocking them keeps content out of training data at the cost of future models knowing less about you. They're also the heaviest crawlers by volume.

AI answer fetchers fire when a real user, mid-conversation, asks something your page can answer. These are the closest thing bot traffic has to warm leads - blocking them removes you from live AI answers.

AI search crawlers build the indexes behind ChatGPT search, Perplexity, and friends. Blocking them removes you from AI search results, the same way blocking Googlebot removes you from Google.

Training-control tokens (Google-Extended, Applebot-Extended) are a special case: they aren't separate crawlers you'll see in logs. They're robots.txt switches that tell an existing crawler's data pipeline "don't use this for AI training." You control them in robots.txt, but you'll never see them visit.

A sane default robots.txt for AI bots

Policies differ, but the most common deliberate setup we see allows answering and search while opting out of training:

# Live AI answers - allow (real users asking about you)
User-agent: ChatGPT-User
Allow: /

User-agent: Perplexity-User
Allow: /

# AI search indexes - allow (this is discoverability)
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Model training - opt out
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Google-Extended
Disallow: /

Adapt to your own position - a content business and a SaaS docs site will reasonably choose opposite training policies. Build yours with the robots.txt generator, and remember the OpenAI trio has three different jobs that deserve three different decisions.

What this list can't do for you

robots.txt is a request, not a lock. Well-run crawlers honor it; scrapers ignore it, and some impersonate the famous names on this list to borrow their whitelisting. Identity should be verified, not read off the user-agent string - reverse-DNS checks catch the fakes, as covered in how to verify Googlebot is real.

A list tells you names, not visits. Whether ClaudeBot actually reads your site, which pages Bytespider is hammering, and whether that "GPTBot" is genuine - that's traffic data, not reference data. Server-side tracking answers it in about five minutes of setup; JavaScript analytics never will, because none of these bots execute JavaScript.

Today's list is not next quarter's list. New bots appear constantly (xAI, Mistral, and a long tail of agents shipped crawlers in the last year alone). This is why DidYouSEO classifies bots server-side - the crawler list updates without you touching your site.

FAQ

What user agent does ChatGPT use to crawl websites? Three, for three purposes: GPTBot (training), ChatGPT-User (live fetching during conversations), and OAI-SearchBot (the ChatGPT search index). Each has its own robots.txt token and can be allowed or blocked independently.

How do I block all AI crawlers at once? There's no single wildcard for "AI bots" - robots.txt rules are per user-agent, so you list each bot's token with a Disallow. Be careful with blanket approaches: they usually catch AI search and answer bots too, which is where AI-era discoverability comes from.

Do AI crawlers respect robots.txt? The major operators (OpenAI, Anthropic, Google, Apple, Perplexity) document that they do, typically within a day of the change. Smaller and shadier crawlers vary - which is why verification and actual traffic data beat trust.

Which AI crawler is the most active? It shifts month to month - Cloudflare's 2026 data has shown GPTBot and ClaudeBot trading the lead, with Googlebot still the largest crawler overall. On your own site, the mix depends heavily on your content; the only way to know is to measure.