- Edition
- November 2026
- Checked
- 20 November 2026
- Formats
- PDF, PNG and this page
- Covers
- 15 operators
One company can run three different bots
A crawler is a program that fetches web pages automatically. AI companies run crawlers for three different purposes, and the larger ones use a separately named bot for each. The chart above sorts them.
- Training crawlers collect pages to train language models. GPTBot, ClaudeBot and Meta-ExternalAgent are in this group.
- Search and index crawlers build the index an AI product retrieves from when it answers. OAI-SearchBot, Claude-SearchBot, PerplexityBot and Meta-WebIndexer are in this group, alongside Googlebot and Bingbot.
- User-triggered fetchers retrieve a specific page when a person asks the assistant about it. ChatGPT-User, Claude-User and Perplexity-User are in this group.
A fourth column lists control tokens. Google-Extended and Applebot-Extended are not crawlers. They are names a site can use in its robots.txt file, the text file that tells crawlers which pages they may fetch, to govern whether pages already crawled by Googlebot or Applebot are used for AI training.
Every bot, as a table
| Operator | Training crawler | Search and index crawler | User-triggered fetcher | robots.txt control token |
|---|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User | — |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User | — |
| Google-CloudVertexBot | Googlebot | — | Google-Extended | |
| Microsoft | — | Bingbot | — | — |
| Perplexity | — | PerplexityBot | Perplexity-User | — |
| Meta | Meta-ExternalAgent | Meta-WebIndexer | Meta-ExternalFetcher | — |
| Apple | — | Applebot | — | Applebot-Extended |
| Amazon | Amazonbot | — | — | — |
| ByteDance | Bytespider | — | — | — |
| Mistral AI | — | — | MistralAI-User | — |
| DuckDuckGo | — | — | DuckAssistBot | — |
| Brave | — | Bravebot | — | — |
| Common Crawl | CCBot (open dataset) | — | — | — |
| xAI | Not documented | Not documented | Not documented | — |
| Moonshot, Zhipu, DeepSeek, Tencent | Third-party lists only | Third-party lists only | — | — |
What the chart shows
The split between training and search is now standard
OpenAI, Anthropic and Meta each document separate bots for training and for search. OpenAI states that a site excluded from OAI-SearchBot will not appear in ChatGPT search answers, while excluding GPTBot does not remove it. The two decisions are independent.
Meta now runs an index crawler
Meta-WebIndexer is documented by Meta as improving Meta AI's search results and allowing Meta to cite and link content. It is the visible sign of the index Meta was reported in 2024 to be building.
Some operators document nothing
xAI publishes no crawler name, no address ranges and no robots.txt policy for Grok. Crawler names attributed to Moonshot, Zhipu, DeepSeek and Tencent appear on third-party lists, and no first-party documentation for them was found for this edition. The chart records both facts and does not print the unconfirmed names as if they were official.
Download, print and reuse
- AI Crawler Chart, PDF: one page, US Letter landscape, vector, with fonts embedded.
- AI Crawler Chart, PNG: 2,640 by 2,040 pixels, for slides and documents.
The chart may be printed, shared and reproduced without charge, provided the SearchEngines.net attribution and the edition date remain visible.
Method, and when this chart was last checked
Categories follow Cloudflare's published bot reference, which classes each bot as an AI crawler, an AI search bot or an AI assistant, and each operator's own documentation where it exists. A dash means no bot of that kind is documented for that operator. A name is printed only where the operator or Cloudflare lists it.
This is the November 2026 edition, checked on 20 November 2026. How crawler rules work is covered in the guide to robots.txt and crawler control; this chart describes the bots and does not recommend a policy.
Frequently asked questions
What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects pages that may be used to train OpenAI's models. OAI-SearchBot builds the index that ChatGPT search retrieves from. They are separate bots with separate robots.txt names, and a site can allow one and exclude the other.
Is Google-Extended a crawler?
No. Google-Extended is a robots.txt control token. It does not fetch pages. It governs whether content already crawled by Googlebot may be used for Google's AI model training.
Does Grok have a crawler?
xAI has published no crawler documentation for Grok: no user-agent name, no address ranges and no robots.txt policy. Names such as GrokBot appear only on third-party lists.