--- title: "Most of the AI crawlers hitting this site are not AI crawlers" date: 2026-09-15 tags: [crawlers, verification, logs, measurement] sources: - url: https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot publisher: "Google Search Central" accessed: 2026-09-15 supports: "Forward-confirmed reverse DNS as the documented verification method, and the published Googlebot prefix list." - url: https://platform.openai.com/docs/bots publisher: "OpenAI" accessed: 2026-09-15 supports: "Published prefix lists for GPTBot, OAI-SearchBot and ChatGPT-User." - url: https://www.cloudflare.com/ips-v4 publisher: "Cloudflare" accessed: 2026-09-15 supports: "Edge address ranges, used to exclude proxy-era log rows from the sample." findings: - claim: "Ten of fourteen claimed AI crawler identities failed verification entirely." method: "Five days of Apache access logs with mod_remoteip enabled. Each source address checked against the operator's published prefix list where one exists, otherwise by forward-confirmed reverse DNS." result: "Verified: Googlebot 22/34, PetalBot 19/19, OAI-SearchBot 13/21, Bingbot 5/5. Zero verified: PerplexityBot 0/33, GPTBot 0/13, Google-Extended 0/13, Bytespider 0/17, ChatGPT-User 0/17, Applebot 0/15, CCBot 0/15, Amazonbot 0/13, meta-externalagent 0/19." - claim: "Three addresses each claimed twelve different crawler identities while scanning for secrets." method: "Grouped requests by source address and counted distinct claimed identities, then read the request paths for those addresses." result: "34.48.216.245, 35.227.68.137 and 35.252.65.136 made 533 requests, 217 of them wearing a crawler identity. All three reverse-resolve to bc.googleusercontent.com. Their most requested paths were /.env, /graphql, /api/graphql and /read-document." - claim: "Almost nothing that fetched /llms.txt could be identified." method: "Counted requests for /llms.txt and joined them against the verification verdicts." result: "27 requests. One came from a verified crawler (PetalBot). The other 26 came from addresses carrying no recognised crawler identity." --- import Chart from '../../components/Chart.astro'; This site has no audience. It published nothing until this week, has no inbound links, and its total human readership is me checking that a deploy worked. It still received 293 requests from software identifying itself as an AI crawler in five days. I wanted to know which of those were real. The answer is that most were not — and the ones doing the impersonating were not interested in the content at all. They were looking for my environment file. ## Why a user-agent string proves nothing I did not know until last week that there is a documented way to check this, and that it is an old one. A crawler announces itself in the `User-Agent` header. That header is a string the client chooses. There is no signature on it, no key exchange, nothing to forge. Writing `GPTBot/1.2` into a request costs nothing and requires no relationship with OpenAI. The check that does work is **forward-confirmed reverse DNS**, and it works like a callback: ```bash dig +short -x 66.249.66.1 # → crawl-66-249-66-1.googlebot.com dig +short crawl-66-249-66-1.googlebot.com # → 66.249.66.1 the address we started from ``` Resolve the address to a hostname. Resolve that hostname back to an address. Accept only if you arrive where you started *and* the hostname sits in the operator's domain. A scraper can put anything in a header; it cannot forge the PTR record for an address it does not control. The second half is the part that matters and the part that gets skipped. A PTR record on its own proves nothing, because whoever controls the address can set it to whatever they like. Only the round trip is evidence. If that shape feels familiar, it is the same reflex as checking a webhook signature instead of trusting the payload, or pinning a certificate instead of trusting the name on it. The SEO literature treats crawler identification as string matching. It is authentication, and authentication has known answers. Some operators do it differently. Google and OpenAI publish prefix lists — a JSON file of the address ranges their crawlers use — and expect you to check membership. Where a list exists it is better evidence than DNS, because it is the operator stating the answer directly rather than you inferring it. ## What I did Five days of Apache access logs, 10 to 15 September, from a server I have root on. That last part matters more than it sounds: almost everyone writing about AI crawler behaviour is on managed hosting, reading a third-party dashboard and inferring. The raw log is the only place this information exists. One setup detail nearly invalidated the whole thing. The site sits behind Cloudflare, so every request arrives at my origin from a Cloudflare edge address, not from the client. Apache logged the proxy. Enabling `mod_remoteip` with Cloudflare's published ranges as trusted proxies restores the real client address — and the trusted-ranges part is not optional, because without it anyone reaching the origin directly could forge the header and write fake crawler hits into my own dataset. Rows logged before that fix carry edge addresses and cannot be verified either way. They are excluded here rather than counted as failures, since they are an artefact of my own setup and would otherwise have dominated the numbers. Every remaining source address was then checked against the operator's prefix list where one exists, and by forward-confirmed reverse DNS otherwise. ## What came back Fourteen distinct identities appeared. Four of them survived verification: | Claimed | Requests | Verified | Disconfirmed | |---|---|---|---| | Googlebot | 34 | 22 | 12 | | PetalBot | 19 | 19 | 0 | | OAI-SearchBot | 21 | 13 | 8 | | Bingbot | 5 | 5 | 0 | | ClaudeBot | 59 | 0 | 42 | | PerplexityBot | 33 | 0 | 33 | | meta-externalagent | 19 | 0 | 19 | | Bytespider | 17 | 0 | 17 | | ChatGPT-User | 17 | 0 | 17 | | Applebot | 15 | 0 | 15 | | CCBot | 15 | 0 | 15 | | Amazonbot | 13 | 0 | 13 | | Google-Extended | 13 | 0 | 13 | | GPTBot | 13 | 0 | 13 | Every request claiming to be GPTBot failed. Every request claiming to be PerplexityBot failed. Google-Extended, the agent Google uses for AI training data and the one publishers argue about blocking, appeared thirteen times and verified zero. Meanwhile PetalBot — Huawei's crawler, which nobody writes thinkpieces about — verified nineteen times out of nineteen. ## The three addresses doing most of it Grouping by source address rather than by user-agent changed the picture completely. Three addresses accounted for 533 requests between them, 217 of which wore a crawler identity. Each of the three claimed **twelve different identities** at different moments: Googlebot, GPTBot, ClaudeBot, PerplexityBot, Applebot, Amazonbot, CCBot, Bytespider, ChatGPT-User, OAI-SearchBot, Google-Extended and meta-externalagent. The same address, rotating costumes. All three reverse-resolve into `bc.googleusercontent.com`. They are rented Google Cloud instances. What settles it is not the DNS, though. It is what they asked for: ``` /.env /.env?raw /@fs/../.env?import&raw?? /graphql /api/graphql /v1/graphql /read-document ``` That is not a crawler. No search engine or language model needs my environment file. This is a vulnerability scanner that puts on crawler user-agents, and my guess — and it is a guess — is that it does so because a request claiming to be Googlebot is less likely to be blocked than one claiming nothing. The impersonation is not an attempt to be indexed. It is camouflage. ## The ClaudeBot problem, which is not the same problem Fifty-nine requests claimed to be ClaudeBot. Forty-two came from those three Google Cloud addresses and are disconfirmed along with everything else they sent. The remaining seventeen came from addresses I cannot verify at all, in either direction. Anthropic publishes no prefix list that I could find at any of the URLs I tried. The addresses in question — in `216.73.216.0/22` — return no PTR record, so there is nothing to forward-confirm. Neither documented method applies. This is worth separating carefully, because the tempting move is to lump it in with the failures and report a bigger number. Those seventeen requests are **unknown**, not fake. They may well be genuine. I have no way to find out, and saying otherwise would be exactly the sloppiness this whole exercise is about. I have recorded them under a distinct state in the database — `none`, meaning no verification method was available — rather than folding them into the disconfirmed count. If Anthropic publishes a range list later, those rows can be re-evaluated without re-collecting anything. ## Nobody is reading llms.txt A smaller finding, and the one I expected least. This site serves `/llms.txt`, the proposed convention for giving language models a plain-text map of a site. It received 27 requests in five days. Exactly one came from a verified crawler, and it was PetalBot. The other 26 came from addresses carrying no recognised crawler identity at all. Some are likely the same scanner traffic; `llms.txt` is a well-known path now, and well-known paths get probed. What did the verified crawlers actually fetch? Almost entirely `robots.txt`. Googlebot asked for `robots.txt` nine times, the homepage twice, a stylesheet three times and the favicon twice. OAI-SearchBot asked for `robots.txt` thirteen times and nothing else at all. Thirteen requests for the rules and zero for the content. The verified agents arriving here are checking whether they are allowed in, then leaving. ## The traffic underneath all of this The crawler question is a small slice of what actually arrives. In the same window the site absorbed thousands of requests for `/xmlrpc.php`, `/wp-login.php`, `/wp-admin/` and `/?author=1`. There is no WordPress here and no PHP handler at all — the site is static files — so every one of those returns a 404 or a 403 and costs nothing. But it is a useful reminder of proportion. The AI crawler conversation is loud, and on a small site the AI crawler traffic is a rounding error against automated probing. It also produced an unexpected argument for the static migration I did last week. Those `/?author=1` requests are WordPress user enumeration: on a WordPress install they return the author's username. They are still arriving, 270 times in the window, and now they return a static homepage that contains nothing. ## How to run this yourself The method is more useful than my numbers, so here is the whole of it. **You need the log from the machine that answers the request.** If a CDN sits in front of your site, its cached hits never reach your origin and never appear in your log. If you are on a platform that does not give you raw access logs at all, this experiment is not available to you, and that constraint is most of why so little of this gets published. **Restore the real client address before you collect anything.** Behind a proxy your log records the proxy. On Apache that is `mod_remoteip` with the CDN's published ranges as trusted proxies; nginx calls it `realip`. Set the trusted ranges explicitly. Without them the header is forgeable by anyone who reaches your origin directly, and you will have built a dataset that an attacker can write to. **Verify in this order.** Check the operator's published prefix list first where one exists — it is the operator answering directly. Fall back to forward-confirmed reverse DNS. Record *which* method produced each verdict, because "failed DNS" and "no method exists" are different claims and collapsing them is how you end up publishing a number you cannot defend. **Cache the verdicts by address, not by request.** Crawler addresses repeat heavily and every lookup is a network round trip. A few hundred lookups covered 70,000 rows here. **Group by address before you draw any conclusion.** This is the step that changed my result completely, and it is one line of SQL. The parser, schema and verification logic behind this post are a few hundred lines of Python with no dependencies beyond the standard library. It runs nightly. I will publish it as a tool once it has survived a few more weeks of its own output. ## What this means if you are doing this work now **Any crawler report built on user-agent strings is wrong.** Not imprecise — wrong, in a specific direction. It will overstate AI crawler interest, because impersonation inflates exactly those numbers and nothing corrects for it. If a tool shows you GPTBot traffic and cannot tell you how it verified it, it is showing you a header. **Blocking by user-agent fails in both directions.** A `robots.txt` rule against GPTBot is honoured by the real GPTBot and ignored by everything wearing its name. You block the agent that was going to behave and inconvenience nobody else. **Group by address before you conclude anything.** Fourteen identities in my logs collapsed to a much smaller number of actors once I grouped by source. The user-agent column invented a crowd. **Publish your uncertain rows as uncertain.** The ClaudeBot seventeen are the most interesting rows in the dataset precisely because I cannot resolve them. ## What remains unknown One site, five days, no audience. The absolute counts here mean nothing — the ratios are the finding, and even those come from a sample small enough that a single scanner shifts them. I cannot tell whether the same ratio holds on a site with real traffic. It may be that impersonation is a fixed background noise, in which case a busy site would see the same absolute volume and a far better ratio. That is a different experiment and it needs a site I do not have. I cannot tell what the three scanners were ultimately for. Reading `.env` requests tells you what they wanted, not who wanted it. I do not know whether the seventeen unverifiable ClaudeBot requests are genuine. I have asked the question the only way I can, which is publicly and with the data attached. And I cannot yet say anything about whether appearing in these logs correlates with appearing in an AI answer. That is the question worth answering, it needs months rather than days, and the log shipping that produced this post is now running nightly so that the version of it I write in January has something behind it.