What Is a Search Engine?

By Bill Hartzer · Updated 19 August 2026

A search engine is three machines, not one

A search engine answers a question about the web by consulting a copy of the web it made earlier. That single sentence contains the whole architecture, and it is the reason search results appear in a fraction of a second rather than in the hours it would take to actually go and look. Three distinct systems have to work for it to be true.

The first is the crawler — an automated program that fetches pages over ordinary HTTP, reads the links inside them, and queues those links to fetch next. Googlebot, bingbot, MojeekBot, Bravebot, SeznamBot, Baiduspider, YandexBot and Naver's Yeti are all crawlers in this sense. Each announces itself in the HTTP request with a user-agent string that names it: Mojeek's crawler sends Mozilla/5.0 (compatible; MojeekBot/0.11; +https://www.mojeek.com/bot.html), and the trailing URL is a documentation page a server administrator can open to find out what just hit their logs.

The second is the index — the engine's own stored, restructured copy of what the crawler fetched. It is not an archive of pages in the form they arrived; it is turned inside out, so that instead of asking "what words are on this page?" the system can ask "which pages contain this word?" and get an answer instantly. Building and holding one at web scale is the expensive part. Perplexity's engineering write-up of September 2025 describes "an exabyte-scale index and crawling apparatus" tracking over 200 billion unique URLs on tens of thousands of CPUs.

The third is the ranking system — the code that takes a query, finds the candidate documents in the index, scores them, and decides which twenty of the several million matches a person sees. This is where engines differ from one another most and disclose least.

The test that separates one search engine from another

The most useful question to ask about any search engine is not how good its results are. It is: whose index is it looking at? The answer is frequently not the one the brand implies, and it changes what the product actually is.

Some engines own the whole stack. Google, Microsoft Bing, Yandex, Baidu, Naver, Seznam, Cốc Cốc, Mojeek, Brave Search and Marginalia all crawl the open web with their own documented crawlers and rank their own indexes. Between them that is a very short list for a planet.

Some engines own none of it. Startpage runs no crawler; it takes Google's results and strips the tracking, and it says so — that is its product, not a deficiency. Yahoo! Search has been served by Microsoft's index since the 2009 alliance. Dogpile, MetaCrawler and WebCrawler are all System1-owned front ends over third-party feeds, which is a particular irony for WebCrawler, since the 1994 original was the first engine to index the full text of the pages it fetched.

And a growing middle group owns part of it. DuckDuckGo operates its own crawlers — DuckDuckBot and DuckAssistBot — and its own indexes for the Instant Answer boxes, but states in its own help pages that it "largely sources" the traditional links from Bing. Kagi crawls with Kagibot into two of its own indexes, Teclis for the web and TinyGem for news, then blends them with anonymised API calls to commercial indexes. Ecosia and Qwant jointly own European Search Perspective, whose index went live in August 2025 and now serves a minority share of their queries alongside Bing and Google feeds.

None of these arrangements is dishonest. But "an alternative to Google" means something entirely different depending on which of the three groups an engine belongs to, and it is the fact that engines advertise least clearly.

What a search engine is not

Several things get called search engines that are structurally something else, and the distinctions are not pedantic — they determine what the tool can and cannot find.

A browser is not a search engine. Chrome, Safari, Edge and Firefox are programs for displaying pages. Each ships with a default engine wired into its address bar, and that default is a commercial arrangement, not a technical property of the browser.

A directory is not a search engine. DMOZ, the Open Directory Project, was a hand-built taxonomy maintained by volunteer editors: a human decided each site's category and wrote its description. It closed in March 2017. Directories search a catalogue of human summaries, not the text of the web.

A submission list is not a crawl. ALIWEB, announced in late 1993 by Martijn Koster and presented at the first World Wide Web conference at CERN in May 1994, never fetched a page. Site owners wrote a small structured file describing their own resources and registered its location; ALIWEB merged those files and searched them. Koster designed it that way deliberately, on the argument that crawlers imposed unacceptable load on servers.

A file-name catalogue is not a web search engine. Archie, which went live on 10 September 1990, catalogued the filenames and directory paths on public anonymous-FTP servers. It read nothing inside those files and ranked nothing. It also predates the public web, which is why the accurate claim for it is "first internet search engine", not "first web search engine".

A computational engine is not an index of the web. Wolfram Alpha answers by calculating over curated structured data. Ask it something no dataset covers and it has nothing to fall back on, because there is no crawl underneath.

Metasearch: a results page assembled from other results pages

A metasearch engine holds no index at all. It forwards a query to several engines at once, collects what comes back, merges the lists and presents one page. The 1990s produced a whole generation of them — MetaCrawler in 1995, SavvySearch, ProFusion, Mamma, Dogpile, Highway 61, Inference Find — at a moment when no single engine covered much of the web and querying four at once genuinely improved recall.

The modern descendant is SearXNG, a free, self-hostable metasearch package that queries configured engines on the user's behalf. Its privacy property is real and structural: the upstream engines see the SearXNG instance, not the person. Its coverage property is equally structural, in the other direction: it can only return what its configured sources return, and it degrades whenever one of them changes terms or blocks it.

The distinction that matters is that metasearch cannot make a result exist. If Bing and Google both lack a page, no interface over them will find it. That is why the presence or absence of an independent index is the load-bearing fact about the search market — merging existing rankings adds interface, never coverage.

Answer engines and the AI layer

Since 2023 a fourth kind of product has grown up alongside the ranked list. An answer engine retrieves documents the same way a search engine does, then passes them to a language model that writes a synthesised answer with citations, instead of — or above — a list of links.

Perplexity is the clearest example: it runs PerplexityBot, holds its own index, and has no non-AI mode. ChatGPT Search runs OAI-SearchBot into OpenAI's own retrieval system while also, by OpenAI's own statement, using third-party search providers. Google's AI Overviews and AI Mode are generated by Gemini models grounded on Google's existing index rather than by a separate crawl. Brave has offered a Summarizer above its results since March 2023, and DuckDuckGo's DuckAssist works the same way, with DuckAssistBot fetching pages in real time to ground the answer.

Underneath, the machinery is the same three-part stack. What changes is the output format — and the click. An answer engine that summarises a page well removes the reason to visit it, which is the substance of the current argument between publishers and AI companies. It is worth noticing that at least one operator has refused the category outright: Mojeek published a post titled "Mojeek is Not an Answer Engine" in September 2025, and treats returning links as an editorial position rather than a limitation.

Who pays for it, and what that buys

Crawling and indexing the web is a permanent cost, not a one-off build, because the web changes underneath you. Something has to fund it, and the funding model shapes the results page more than any algorithm does.

Advertising. The dominant model. Google, Bing, Baidu, Yandex, Naver, Seznam and Cốc Cốc all sell placement against queries. It makes search free at the point of use and creates a permanent structural interest in commercially valuable queries.
Syndication and revenue share. Microsoft supplies results and ads to partners and pays a share of the revenue back. This is how Yahoo, DuckDuckGo and for years Ecosia funded themselves without a crawler.
Subscription. Kagi charges the user directly and argues that with no advertiser, the only party the ranking has to satisfy is the person paying. The trade is obvious: no ads, and a paywall.
Metered API access. Increasingly significant. Brave, Mojeek, Perplexity and You.com all sell index access to developers and AI companies. You.com went furthest — it withdrew its consumer search product and now sells its index as an API.
Mission funding. Ecosia spends the large majority of its advertising profit on tree planting and, since October 2018, is held in a steward-ownership structure that means it cannot be sold or pay dividends. Marginalia is run by one person in Sweden on a scale of hosting costs that a hobby budget covers.

Why so few organisations run one

The barrier is no longer purely technical. Marginalia demonstrates that: one engineer, a custom Java stack, a published crawler user-agent of search.marginalia.nu, a static IP range, and hundreds of millions of documents. Independent crawling at meaningful scale is possible.

What has changed is access. In 2024 Reddit blocked non-Google crawlers, cutting Mojeek and other independent engines off from a large body of content while Google retained access under a paid arrangement. That is the clearest available illustration of how index consolidation now entrenches itself: not through better crawling technology, but through content licensing that only the largest incumbents can afford. An engine can build a perfectly good crawler and still be unable to see a material fraction of the web.

The same pressure runs the other way too. Microsoft retired the Bing Search APIs entirely on 11 August 2025, which forced every product that had been quietly reselling Bing to either own an index or find another supplier. Products built on someone else's index are, by construction, one contract change away from having no product.

Frequently asked questions

What is a search engine in simple terms?

A search engine is a system that stores a copy of the web and searches that copy. A crawler fetches pages and follows their links; an indexer turns the text into a structure that can be queried instantly; a ranking system scores the matches and orders them. When you search, you are searching the engine's stored index, not the live web.

What are the three main parts of a search engine?

Crawling, indexing and ranking. Crawling is the automated fetching of pages by a program such as Googlebot, bingbot or MojeekBot. Indexing is the processing and storage of what was fetched into a queryable structure. Ranking is the scoring that decides the order results appear in. Engines differ most in the third, and disclose least about it.

How many search engines have their own index?

Far fewer than the number of search engines. Google, Microsoft Bing, Yandex, Baidu, Naver, Seznam, Cốc Cốc, Mojeek, Brave Search, Marginalia and Perplexity all crawl the open web and rank their own indexes. Most other named engines re-serve one of those indexes, blend several, or sit somewhere in between.

Is a browser a search engine?

No. A browser — Chrome, Safari, Edge, Firefox — is a program that displays pages. A search engine is a separate service the browser sends a query to. Every browser ships with a default engine wired into its address bar, and that default is a commercial arrangement rather than a technical necessity; it can be changed.

What is the difference between a search engine and a directory?

A directory is a catalogue of sites organised and described by people. DMOZ, the Open Directory Project, was the largest example; volunteer editors placed each site in a category and wrote its summary, and it closed in March 2017. A search engine builds its catalogue automatically from the text of pages, with no human deciding what belongs where.

Is DuckDuckGo its own search engine?

Partly. DuckDuckGo operates its own crawlers, DuckDuckBot and DuckAssistBot, and its own indexes behind the Instant Answer boxes, but its own help pages state that the traditional web links are largely sourced from Bing. It is neither an independent index nor a plain Bing skin — it is a hybrid, and both simplified descriptions of it are wrong.

What is the difference between a search engine and an answer engine?

A search engine returns a ranked list of links and leaves the reading to you. An answer engine retrieves the same kind of documents and then has a language model write a synthesised answer, usually with citations. Perplexity and ChatGPT Search are answer engines; Google's AI Overviews add an answer layer above a still-ranked list of links.

Do search engines index the whole web?

No, and Google says so openly: it crawls far more than it keeps, and discards pages it judges low value. Every index is a selected subset, chosen under a finite crawl and storage budget. Two engines can both be working correctly and still disagree about whether a page exists, because neither holds all of it.

What is a search engine?