How to Verify Googlebot Is Really Googlebot (and Catch the Impostors)

Here's an uncomfortable fact about your server logs: the user-agent string is just text, and anyone can send any text they like. When a request says it's Googlebot, that's a claim, not a credential. Scrapers impersonate Googlebot constantly - because most sites give the famous crawlers a free pass through rate limits and paywalls.

The good news: you can verify Googlebot definitively in about ten seconds. This post shows the official method, the equivalent checks for Bing and Apple, and how to spot the fakes in your own traffic.

Why impostors pretend to be Googlebot

Impersonating a search crawler is the scraper's cheapest trick, and it works because sites want Googlebot to crawl freely:

• Rate limits often whitelist "Googlebot" by user-agent
• Paywalls and gated content frequently serve full content to search crawlers
• Bot-blocking rules almost never touch anything claiming to be Google

With bots now generating the majority of web traffic, the volume hiding behind famous names is not a rounding error - it's a standing category of abuse.

How to verify Googlebot: the official reverse-DNS method

Google doesn't publish a simple list you can memorize; it publishes an official verification procedure built on reverse DNS. The logic:

1. Take the visiting IP address from your logs
2. Run a reverse DNS lookup - a real Googlebot IP resolves to a hostname ending in googlebot.com or google.com
3. Run a forward DNS lookup on that hostname - it must resolve back to the same IP

Both directions must match. A scraper can fake the user-agent, but it can't make Google's DNS vouch for its IP.

# Step 1+2: reverse lookup
$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

# Step 3: forward lookup must return the same IP
$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Google also publishes machine-readable JSON lists of its crawler IP ranges, which is what automated verification uses instead of running DNS queries per request.

Verifying the other big crawlers

Bingbot works the same way: Microsoft documents reverse-DNS verification with hostnames ending in search.msn.com, plus a published IP list and even a web-based "Verify Bingbot" tool.

Applebot verifies identically - reverse DNS to an applebot.apple.com hostname, per Apple's Applebot documentation.

AI crawlers are messier. OpenAI and Anthropic publish IP ranges rather than DNS verification, and some smaller AI bots publish nothing at all. Cloudflare maintains a verified-bots directory that tracks which crawlers can actually be validated - worth checking before you whitelist anything by name.

What a verification result actually tells you

The failed bucket is the interesting one. A "Googlebot" visit that fails reverse DNS isn't Google having a bad day - the check doesn't produce false positives. It's someone wearing a costume.

Doing this automatically instead of by hand

Running host commands against every suspicious log line doesn't scale past curiosity. This is exactly why we built IP verification into DidYouSEO's bot traffic tracker: every visit claiming to be Googlebot, Bingbot, or Applebot gets the reverse-DNS check automatically, and the dashboard shows a shield icon per visit - verified, failed, or unchecked. Setup takes about five minutes.

Whether you use our tool or a log-analysis script, the principle stands: any bot-related decision - whitelisting, blocking, crawl-budget analysis - should run on verified identity, not user-agent strings.

Common mistakes

Whitelisting by user-agent alone. A firewall rule that says "never rate-limit Googlebot" is an open invitation - scrapers know exactly which magic words to send.

Blocking a real crawler by IP reputation. The reverse is also common: an overzealous IP blocklist that catches genuine Google ranges. That quietly kills your crawl coverage, and it looks identical to a technical SEO problem when rankings slide - our technical SEO audit checklist covers how to spot crawl-access issues.

Verifying once and caching forever. IPs get reassigned. Verification results are safe to cache for days, not years.

FAQ

Can a scraper fake reverse DNS? Not meaningfully. The scraper would need to control the reverse DNS of its own IP and have Google's forward DNS point that hostname back at its IP - the second half is impossible without being Google.

Does a failed verification ever happen to real Googlebot? The documented method doesn't produce false positives when both lookups are performed. Failures mean an impostor - or, occasionally, a user-triggered Google fetch (like Site Verifier) that has different verification hostnames than the main crawler.

How common are fake Googlebots? Common enough that Google built an official verification page and major CDNs run verification on every request. On sites we track, impostor rates vary wildly - which is itself the point: you don't know your number until you check.

Do I need to verify AI crawlers like GPTBot too? Where possible, yes - OpenAI publishes IP ranges for its bots. Knowing which OpenAI bot is which matters first, though: the right response differs per bot even when they're genuine.