SearchEngines.net logo — an independent reference on search enginesWho runs which index

The shift to AI search

AI crawler chart

Which AI bots collect pages for training, which build a search index, and which fetch a page only when a user asks.

AI Crawler ChartWHICH BOT FEEDS WHAT · NOVEMBER 2026 EDITIONOPERATORTRAINING CRAWLERSEARCH AND INDEX CRAWLERUSER-TRIGGERED FETCHERROBOTS.TXT TOKENOpenAIGPTBotOAI-SearchBotChatGPT-User—AnthropicClaudeBotClaude-SearchBotClaude-User—GoogleGoogle-CloudVertexBotGooglebot—Google-ExtendedMicrosoft—Bingbot——Perplexity—PerplexityBotPerplexity-User—MetaMeta-ExternalAgentMeta-WebIndexerMeta-ExternalFetcher—Apple—Applebot—Applebot-ExtendedAmazonAmazonbot———ByteDanceBytespider———Mistral AI——MistralAI-User—DuckDuckGo——DuckAssistBot—Brave—Bravebot——Common CrawlCCBot (open dataset)———xAINot documentedNot documentedNot documented—Moonshot, Zhipu, DeepSeek, TencentThird-party lists onlyThird-party lists only——Collects pages for model trainingBuilds the index AI answers citeFetches one page when a user asksA robots.txt token, not a crawlerCategories follow Cloudflare's bot reference and each operator's own documentation. A dash means no crawler of that kind is documented.Sources and method: www.searchengines.net/guides/ai-crawler-chart/© 2026 SearchEngines.net · Free to print and share with attribution
November 2026 edition. Download the printable PDF or the PNG image.
Edition
November 2026
Checked
20 November 2026
Formats
PDF, PNG and this page
Covers
15 operators

One company can run three different bots

A crawler is a program that fetches web pages automatically. AI companies run crawlers for three different purposes, and the larger ones use a separately named bot for each. The chart above sorts them.

  • Training crawlers collect pages to train language models. GPTBot, ClaudeBot and Meta-ExternalAgent are in this group.
  • Search and index crawlers build the index an AI product retrieves from when it answers. OAI-SearchBot, Claude-SearchBot, PerplexityBot and Meta-WebIndexer are in this group, alongside Googlebot and Bingbot.
  • User-triggered fetchers retrieve a specific page when a person asks the assistant about it. ChatGPT-User, Claude-User and Perplexity-User are in this group.

A fourth column lists control tokens. Google-Extended and Applebot-Extended are not crawlers. They are names a site can use in its robots.txt file, the text file that tells crawlers which pages they may fetch, to govern whether pages already crawled by Googlebot or Applebot are used for AI training.

Every bot, as a table

AI crawlers by operator and purpose, November 2026
OperatorTraining crawlerSearch and index crawlerUser-triggered fetcherrobots.txt control token
OpenAIGPTBotOAI-SearchBotChatGPT-User—
AnthropicClaudeBotClaude-SearchBotClaude-User—
GoogleGoogle-CloudVertexBotGooglebot—Google-Extended
Microsoft—Bingbot——
Perplexity—PerplexityBotPerplexity-User—
MetaMeta-ExternalAgentMeta-WebIndexerMeta-ExternalFetcher—
Apple—Applebot—Applebot-Extended
AmazonAmazonbot———
ByteDanceBytespider———
Mistral AI——MistralAI-User—
DuckDuckGo——DuckAssistBot—
Brave—Bravebot——
Common CrawlCCBot (open dataset)———
xAINot documentedNot documentedNot documented—
Moonshot, Zhipu, DeepSeek, TencentThird-party lists onlyThird-party lists only——

What the chart shows

The split between training and search is now standard

OpenAI, Anthropic and Meta each document separate bots for training and for search. OpenAI states that a site excluded from OAI-SearchBot will not appear in ChatGPT search answers, while excluding GPTBot does not remove it. The two decisions are independent.

Meta now runs an index crawler

Meta-WebIndexer is documented by Meta as improving Meta AI's search results and allowing Meta to cite and link content. It is the visible sign of the index Meta was reported in 2024 to be building.

Some operators document nothing

xAI publishes no crawler name, no address ranges and no robots.txt policy for Grok. Crawler names attributed to Moonshot, Zhipu, DeepSeek and Tencent appear on third-party lists, and no first-party documentation for them was found for this edition. The chart records both facts and does not print the unconfirmed names as if they were official.

Download, print and reuse

The chart may be printed, shared and reproduced without charge, provided the SearchEngines.net attribution and the edition date remain visible.

Method, and when this chart was last checked

Categories follow Cloudflare's published bot reference, which classes each bot as an AI crawler, an AI search bot or an AI assistant, and each operator's own documentation where it exists. A dash means no bot of that kind is documented for that operator. A name is printed only where the operator or Cloudflare lists it.

This is the November 2026 edition, checked on 20 November 2026. How crawler rules work is covered in the guide to robots.txt and crawler control; this chart describes the bots and does not recommend a policy.

Frequently asked questions

What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects pages that may be used to train OpenAI's models. OAI-SearchBot builds the index that ChatGPT search retrieves from. They are separate bots with separate robots.txt names, and a site can allow one and exclude the other.

Is Google-Extended a crawler?

No. Google-Extended is a robots.txt control token. It does not fetch pages. It governs whether content already crawled by Googlebot may be used for Google's AI model training.

Does Grok have a crawler?

xAI has published no crawler documentation for Grok: no user-agent name, no address ranges and no robots.txt policy. Names such as GrokBot appear only on third-party lists.

Sources

Top