Guides · AI crawlers

AI Crawlers in 2026: The Full List (GPTBot, ClaudeBot, CCBot, PerplexityBot)

Last verified: October 2026 · About 10 minutes of reading

Eight tokens have public vendor documentation and a stated purpose, and they fall into two groups with opposite consequences. This page lists them with their owners, what they fetch, and what their vendors actually promise about robots.txt — which for one of them is nothing. It also explains why you cannot identify a real crawler from its user agent, the detail most lists leave out.

The two groups that matter

Two groups matter: training crawlers that collect material for future models, and search crawlers that fetch pages to answer a question now.

Blocking the first group limits what future models learn from you. Blocking the second removes you from answers today. Treating them as one category is the single most common policy mistake, because the two decisions have different costs and different reversibility.

The list below is the set evaluated by Pingory's AI Readability check. It is deliberately short: these are the tokens with public vendor documentation and a stated purpose, not every bot that has ever requested a page.

One clarification before the tables. A user agent is a string the client sends. A token is the name you write in robots.txt. For seven of the eight entries those are the same string, which is convenient but also the source of the Google-Extended confusion covered below. The address a request comes from is a third thing entirely, and it is the only one that carries evidence.

Training crawlers

The training group is GPTBot, ClaudeBot, Google-Extended and CCBot, and only three of them are actual fetchers.

Two notes on that table. First, Google-Extended has no standalone user agent: it is a control token you disallow in robots.txt, and the fetching is done by Googlebot under Google's normal crawl rules. Disallowing it does not remove you from Google Search. Second, CCBot feeds a corpus rather than a single product, so blocking it affects every downstream consumer of Common Crawl, not just one vendor.

Search crawlers

The search group mixes two scheduled index crawlers (OAI-SearchBot, PerplexityBot) with two user-triggered fetchers (ChatGPT-User, Claude-User).

The distinction inside this group is frequency. Index crawlers arrive on a schedule and can be managed like a search engine. User-triggered fetchers arrive when a person is already looking at your domain, so a block produces a visible failure rather than a quiet omission. If a customer pastes your URL into a chat product and gets "I cannot access that page", this group is usually why.

One member of the group breaks the pattern. OpenAI states that robots.txt rules may not apply to ChatGPT-User, because the request is initiated by a user rather than by a crawl schedule. A rule written for that token is a statement of preference, not a control. If you genuinely need to stop it, the block has to happen at the edge — a WAF rule, authentication, or a rate limit — and even then you are matching a user agent that can be copied.

It also means the two halves of this group fail differently. Block OAI-SearchBot and your pages quietly drop out of an index you never see. Block ChatGPT-User and the failure happens in front of a person, in a conversation, with no error message reaching you. Neither shows up in your analytics as a 403, which is why the check has to happen on the policy side.

Do AI crawlers respect robots.txt?

Seven of the eight tokens are documented to obey robots.txt; the eighth, ChatGPT-User, is explicitly outside that promise.

robots.txt is defined by RFC 9309, and each vendor in the tables above documents support for its token — with the ChatGPT-User exception noted. That is the accurate wording: documented to obey, or declares support. There is no authority that audits compliance, no penalty for ignoring a rule, and no technical mechanism that forces a fetch to stop. A crawler operator can change behaviour without notice, and a new crawler can appear with no policy at all.

What you can rely on is narrower and still useful. Because the vendors publish the tokens, a rule written against those tokens is legible to the operator. If a documented crawler ignores an explicit rule, that is a fact worth recording — in your logs, with a timestamp and the source IP — rather than an assumption either way.

This is also why a checker should describe your policy rather than simulate a bot. If a tool sends a fake GPTBot header and your WAF answers 403, the only thing proven is that the WAF blocks forged requests. Real crawlers arrive from vendor addresses, so the simulation can report a block that does not exist.

Google-Extended is a token, not a crawler

This is the item most lists get wrong. Google-Extended does not appear as a user agent in your logs, because there is no Google-Extended fetcher. It is a robots.txt token that tells Google whether content already crawled by Googlebot may be used for Gemini training and grounding. Two consequences follow:

• You will never see a request from it, so "I checked my logs and Google-Extended never visits" proves nothing.
• Disallowing it is a usage decision, not an access decision. Googlebot keeps crawling; you stay in Search.

For that reason the Pingory checker evaluates Google-Extended as policy rather than as traffic, and reports it separately from the fetchers.

How to tell a real crawler from a spoofed one

A user agent is self-declared text, so only the source IP address can support a claim about identity.

Anyone can send User-Agent: GPTBot from a laptop. That is why networks such as Cloudflare check the connecting address against a published list of vendor ranges before treating a request as legitimate, and why a user-agent match alone is not evidence.

To check a suspicious request in your own logs, work in this order:

1. Log the address, not the string. Keep the client IP for at least a few days; user agents are trivially forged and useless alone.
2. Reverse-resolve it. A genuine vendor address usually resolves to a hostname in that vendor's domain. Forward-confirm the result to make sure the name points back to the same IP.
3. Compare against published ranges. OpenAI, Google and Common Crawl publish their ranges. Check membership, not similarity.
4. Do not test by impersonation. Sending a fake GPTBot header from your own machine tells you how your WAF treats a forged request. It tells you nothing about whether the real crawler can reach you.

Cloudflare users get part of this for free: verified bot detection compares the connecting address with published ranges and labels the request accordingly in firewall analytics. If your logs come from a proxy that strips the client address, no amount of user-agent analysis will recover the missing evidence. Fix the logging before drawing conclusions from it.

What to do with this list

Write rules per token, starting from the policy you want: allow retrieval, decide separately about training, then verify the result.

The mainstream 2026 position is to allow the search group and make a deliberate choice about the training group. That combination is not a failure state; it is a policy. What matters is that the decision is explicit and that the file actually implements it.

• If you want a chance to appear in AI answers, keep the three robots-controlled search tokens allowed on the pages that answer customer questions. That buys eligibility, not a citation: the answer still depends on the index, the question and the model.
• If you want to limit future training, disallow the training group — and remember that Google-Extended is a token, so the rule looks different from the others.
• If you are unsure, change nothing for a week and read your logs first. Blocking is reversible; a lost answer is not visible to you at all.

Concrete syntax for each token, plus Cloudflare, Vercel and nginx versions, is in how to block AI crawlers without breaking Google. If you only want to know where you stand right now, run the free AI readability check: it reports allow or block for all eight tokens on both the homepage and a deeper path, and it never impersonates a crawler to do it. The background on the file that pairs with these rules is in what llms.txt is.

Whatever you choose, write down the date and the reason. Policies drift: a CDN default changes, a security plugin ships a new rule set, a staging robots.txt gets promoted to production. A one-line note in the repository beats reconstructing the decision six months later from firewall logs.

Last verified: October 2026. Token names, owners and documented behaviour were checked on 2026-10-24 against each vendor's crawler documentation.