An AI search engine is not a model that has memorised the web. It is a retrieval system with a writer bolted to the front. Understanding that order — retrieve first, generate second — explains almost everything about how these products behave, including what they get wrong.
What actually happens when you ask one a question
A language model's knowledge is frozen at the point its training finished, and it has no way to check anything. To answer a question about the world as it is now, the product has to go and get pages. The sequence is roughly: the system rewrites your question into one or more search queries; those queries are sent to an index; a set of candidate documents comes back; some of them are fetched in full; and the model then writes prose over that material, attaching citations. This is usually called grounding. Everything the answer knows about the present moment arrived through that retrieval step.
Two consequences follow, and both are underappreciated. First, the query you typed is not the query that was run. The reformulation step is invisible; Microsoft's own documentation for Copilot describes it plainly — Copilot "generates a search query that it sends via secure connection to the Bing search service." You cannot see or edit that query, and a bad reformulation produces a confident answer built on the wrong documents. Second, whoever owns the index decides what the model is allowed to see. The generation layer is impressive and increasingly commoditised. Retrieval is the part that is expensive, defensible, and usually bought in.
Grounded on whose index?
The four products in this category sit at four different points on that question, and the differences are larger than the interfaces suggest.
- Microsoft Copilot has no crawler and no index of its own. There is no CopilotBot; publishers control their appearance in Copilot answers by managing
bingbot. Copilot writes a query, sends it to the Bing search service, and paraphrases what comes back. Its substrate is a first-party index — Microsoft owns Bing — but Copilot itself does not rank the web; Bing ranks, and Copilot selects from the top of Bing's answer. Microsoft's documentation also notes that web grounding is a toggle, and that with it off the answers come from model weights alone, with a knowledge cutoff. - ChatGPT Search is the most complicated stack of the four, and OpenAI's own documentation understates it. OpenAI runs
OAI-SearchBot, a crawler that feeds an internal retrieval system, and separately buys web results from third parties — its launch post said the product "leverages third-party search providers, as well as content provided directly by our partners." Independent measurement published in August 2026 found the internal index supplying about three-quarters of results on free-tier instant answers, and about three-quarters of results on the paid "thinking" mode coming instead from scraped Google results purchased from commercial data providers. The same study found only about 1.5% of internal-index URLs appearing in Bing's top twenty for the same query, which is strong evidence that whatever OpenAI is serving from its own index, it is not a Bing passthrough. Treat that as credible third-party research rather than company disclosure. - Perplexity began the other way round. In its early period it was widely reported to be retrieving through a third-party search API, with the language model as the differentiator. By September 2025 it was publishing the engineering of a large crawler and index of its own — an architecture it describes as covering hundreds of billions of web pages, with a multi-stage ranking pipeline built to feed a model rather than to render a results page. Whether particular verticals are still served by partner APIs is not disclosed.
- Kagi is explicitly a blend and says so. It runs a crawler,
Kagibot, feeding two indexes of its own: Teclis, a deliberately non-commercial web index weighted toward personal sites, blogs and forums, and TinyGem for news. For general coverage it makes anonymised API calls into commercial indexes. Its documentation used to name those partners and was generalised in May 2024 to a description rather than a list, so the current supplier mix is not public. In practice the head of a results page on a commercial query is mostly partner material, and the own index surfaces on obscure and non-commercial queries.
The crawler rules changed, and publishers noticed
AI products introduced a distinction that did not exist in classic search: separate crawlers for separate purposes. OpenAI runs one bot for search indexing, a different one for model training, and a third for user-initiated page fetches — and blocking the training crawler does not remove a site from search answers, because that is governed by the search crawler. Both OpenAI and Perplexity document that their user-initiated fetchers may not follow robots.txt, on the reasoning that a request triggered by a specific human question is not a crawl. Publishers dispute that reasoning, and it is the root of most of the ongoing conflict between AI search products and the sites they cite.
How to read this category honestly
The category boundary is soft and getting softer. Bing has served generative answers above conventional results since 2023; Google shows AI overviews on a large share of queries; Yahoo's Scout, launched in January 2026, is an AI answer engine grounded on Bing with a third party's language model. By any functional test several engines filed elsewhere on this site are AI search products too. What keeps these four together is that the written answer is the product rather than a layer on top of one.
The failure modes are worth stating plainly, because the interface actively conceals them. A cited answer is not a verified answer: the citation shows where a claim came from, not that the source said it or was right. Retrieval failures are invisible — if the reformulated query missed, you get a fluent answer built on whatever was returned instead, with no signal that anything went wrong. And these products are structurally opaque in a way link-based engines are not: with ten blue links you can see the sources and judge them yourself, whereas with a generated answer the selection has already been made for you by a system whose supplier list, in three of these four cases, is not fully published.