AI search and the rise of answer engines

What actually changed

On 7 February 2023 Microsoft put a GPT-4-based chatbot on the Bing results page. Google announced its equivalent, then called the Search Generative Experience, in May 2023; AI Overviews reached every US user in May 2024 and more than 100 countries on 28 October 2024, with the conversational AI Mode following in March 2025 and Gemini 3 becoming the default model behind Overviews globally on 27 January 2026. OpenAI launched ChatGPT search on 31 October 2024, opening it to users without an account on 5 February 2025. Perplexity had already shipped the format as its entire product on 7 December 2022.

What changed in those three years is the output. A search engine used to hand you a ranked list of documents and leave the reading to you. An answer engine reads a handful of documents for you and writes a paragraph, with links attached. The question moves from "which pages are about this?" to "what is the answer?", and the page you land on is prose rather than a list.

What did not change is everything underneath. A generated answer still requires that somebody has crawled the web, stored a copy of it, and ranked candidate pages for the query — because the model has to be given documents before it can summarise them. AI search did not replace crawling, indexing and ranking. It added a layer on top and reduced how much of the page the ranked list gets to occupy.

Retrieval and generation are two different systems

The most useful mental model is to separate the two halves, because they are built by different teams, often by different companies, and they fail in different ways.

Retrieval finds documents. It is ordinary search: an index is queried, candidates are scored, a shortlist comes back. Generation is a language model writing sentences, conditioned on that shortlist. The industry term for feeding retrieved documents into a model is grounding, and the reason it exists is that a model without grounding answers from its training data, which has a cutoff date and no way to tell you it is out of date.

Two consequences follow, and both are counter-intuitive. First, a good answer engine can be built by a company with no index at all, provided it can buy retrieval from someone who has one. Second, the retrieval step is usually invisible. Microsoft's own documentation describes Copilot parsing your prompt, generating its own search query, and sending that to the Bing search service — so the answer depends on a query you never typed and cannot inspect or edit. Microsoft also notes that web grounding can be switched off entirely, in which case the answer comes from model weights alone and can look identical to a live one.

The failure modes differ accordingly. Retrieval failures give you a confident summary of the wrong documents. Generation failures give you a fluent sentence the retrieved documents do not actually support.

Who actually owns the index behind the answer

This is where the AI search market divides, and the answers are not the ones most coverage assumes.

• Google grounds AI Overviews and AI Mode on its own index, using what it describes as a query fan-out technique. No separate crawl is involved.
• Microsoft Copilot has no crawler and no index. It grounds on Bing. That is not a criticism of it, but it means Copilot cannot be a second opinion on a Bing result — it is the same index with fluency added.
• Perplexity built its own. It was, in substance, a Bing API reseller with a summarisation layer for roughly its first two years; by September 2025 it was describing an index tracking over 200 billion unique URLs, and it kept working when Microsoft retired the Bing Search APIs on 11 August 2025, which a reseller could not have done.
• OpenAI runs OAI-SearchBot into its own retrieval system, and also buys results. Independent measurement published in Search Engine Land on 17 August 2026 found OpenAI's own index supplying about 75% of free-tier results, only 1.5% URL overlap with Bing's top 20, and roughly 75% of paid-tier "thinking" results coming from scraped Google data purchased through third-party providers. OpenAI's own help pages say only that it "sometimes partners with other search providers", naming Bing and Shopify.
• Yahoo Scout, launched 27 January 2026, is grounded on Bing's index with Anthropic's model on top — Yahoo's own description is that the index is Bing's and the ranking and experience are Yahoo's.
• Kagi and DuckDuckGo both put AI over their existing blended retrieval, and both make it optional.

The pattern is that AI has not reduced the number of web indexes in the world. If anything it has increased it, because grounding data turned out to be the scarce input — which is precisely why Perplexity and OpenAI built crawlers, and why Brave, Mojeek and Perplexity now all sell index access as a product.

The new crawlers, and the distinction publishers keep getting wrong

AI search brought a new generation of bots, and their documentation is worth reading closely because the names do different jobs.

OpenAI runs at least four. OAI-SearchBot is the search-index crawler; OpenAI states that sites opted out of it will not appear in ChatGPT search answers. GPTBot is the model-training crawler. ChatGPT-User fetches a page when a user asks the assistant to look at it, and OpenAI's documentation says robots.txt rules may not apply to it. OAI-AdsBot arrived with the advertising product. The separation of search visibility from training consent is genuinely useful design, and it is also the thing most commonly got backwards: blocking GPTBot governs training, not appearance in answers.

Google's equivalent is Google-Extended, which is a robots.txt token with no user-agent of its own; it controls whether a site's content is used to train and ground Gemini models, and Google states it does not affect inclusion in Search.

Perplexity documents two: PerplexityBot, the indexing crawler, and Perplexity-User, which fetches a page in response to a live question and which Perplexity's own documentation says generally ignores robots.txt, on the reasoning that the request is user-initiated. Publishers dispute that reasoning, and it is the root of much of the conflict around the company. DuckDuckGo's DuckAssistBot takes the opposite line: it honours robots.txt with changes taking effect within 72 hours, and DuckDuckGo states the fetched data is not used to train models.

Tiered truth, and the things you cannot inspect

A ranked list of ten links is an auditable object. You can see what was chosen, in what order, and go and check. A paragraph of generated prose is not, and three specific opacities follow from that.

The query is rewritten. Copilot writes its own Bing query from your prompt. ChatGPT rewrites queries before sending them to providers. In both cases the retrieved set depends on words you did not choose and cannot see.

Different users get different indexes. The Search Engine Land measurement found free and paying ChatGPT users being served from materially different retrieval backends — the cheap internal index for one, expensive purchased Google-derived data for the other. That is not a quality tier in the ordinary sense; it is a different view of the web depending on what you pay, and it is not disclosed in the interface.

There is no results list to disagree with. On a conventional engine, a bad top result is visibly a bad top result and the second result is right there. An answer engine has already made the selection and folded it into a sentence. Perplexity has no non-AI mode at all; ChatGPT decides for itself when to retrieve.

None of this makes the answers wrong. It makes them harder to check, which is a different and slower kind of problem.

What a citation actually proves

Inline numbered citations were Perplexity's core contribution to the format and everyone copied them, correctly — an answer with sources is better than an answer without. But a citation is a claim about provenance, not a guarantee of accuracy. It says a document was retrieved. It does not say the sentence it is attached to reflects what that document says.

The evidence for treating this seriously is specific. Forbes and Wired both published accounts in June 2024 alleging that Perplexity summarised articles it had not properly accessed and attributed material it should not have; the company faces copyright suits from News Corp, Reddit and three Japanese publishers. On the measurement side, the August 2026 study of ChatGPT's retrieval stack found that pages the assistant actually opened were cited 74% of the time, while pages merely pulled into the candidate set were cited 7% of the time — reassuring in one direction, and a reminder in the other that the citation list and the reading list are not the same list.

The practical implication is unglamorous: on anything that matters, open the links. A generated answer is a research assistant's first draft, and the citations exist so that the draft can be checked rather than so that it need not be.

The money arrives

The structural critique of advertising-funded search has always been that the incentive to serve the reader and the incentive to serve the buyer diverge. AI answers were, briefly, an escape from that. They are not any more.

OpenAI announced advertising in ChatGPT on 17 January 2026, targeted at free-tier adult users in the United States, with ads appearing by March 2026 — corroborated independently by the existence of OAI-AdsBot in its crawler documentation. That is the largest AI answer surface on the internet acquiring an advertiser, on a page with no visible ranking to inspect and no ads column to skip past. Auditing an advertiser's influence on a paragraph of prose is considerably harder than auditing its influence on a blue link.

Perplexity ran the experiment in reverse. It launched AI-integrated ad formats in late 2024 and discontinued them in February 2026, moving to a subscription-first model — a notable reversal for the flagship independent answer engine, and a sign that nobody has yet settled how this is paid for.

Money also flows the other way. OpenAI has signed content-licensing deals with publishers including the Associated Press, Axel Springer, Condé Nast, the Financial Times, Le Monde, News Corp, Reuters and The Atlantic, and Perplexity runs a publisher revenue-sharing programme introduced in response to complaints. The unresolved question underneath all of it is what happens to the open web if answers stop sending clicks to the sites the answers are made from.

Turning it off, and what has not changed

The option to have search without generated answers is unevenly distributed, and the differences are documented rather than a matter of preference.

• DuckDuckGo publishes a dedicated AI-free endpoint, noai.duckduckgo.com, which it describes as turning off all AI features and filtering AI-generated images with no settings required, alongside per-feature toggles. It is the most straightforward opt-out any mainstream engine offers.
• Kagi treats AI as a feature you invoke rather than a layer applied to every page; its Quick Answer does not fire on an ordinary query.
• Mojeek declined to become an answer engine at all and said so publicly in September 2025.
• Marginalia has no AI mode, deliberately — it is a keyword retrieval engine by conviction.
• Google has no account-level switch for AI Overviews. There is a Web filter, added in May 2024, that returns plain links for a single search, but it is a per-query workaround rather than a setting.

Underneath the argument, the machinery is the same machinery. Somebody still has to run a crawler that respects — or ignores — robots.txt. Somebody still has to store a copy of the web and keep it fresh. Somebody still has to rank candidate documents before a model can be handed anything to read. The number of organisations doing that work has gone up slightly since 2022, not down, and the newest entrants built crawlers for exactly the same reason the old ones did: you cannot answer questions about the web without first having a copy of it.

Frequently asked questions

Sources

• openai.com/index/introducing-chatgpt-search/
• developers.openai.com/api/docs/bots
• searchengineland.com/chatgpt-retrieval-stack-index-cache-pages-485036
• docs.perplexity.ai/guides/bots
• learn.microsoft.com/en-us/copilot/privacy-and-protections
• duckduckgo.com/duckduckgo-help-pages/ai-features/about-noaiduckduckgocom
• developers.google.com/search/docs/crawling-indexing/google-common-crawlers
• research.perplexity.ai/articles/architecting-and-evaluating-an-ai-first-sear…