How search engines work

A search engine is a stored copy of the web, plus an opinion about it

The single most useful thing to understand about search is that a search engine does not search the internet when you type a query. It searches its own index — a stored, pre-processed copy of the pages it has already visited, held on its own servers. Your query never travels out across the web: it goes to one company's data centre, is matched against that company's copy of the web, and comes back ordered by that company's judgement about which pages are worth showing first.

That is why two search engines can return completely different answers to the same question. They are not looking at the same web — they are looking at different copies of it, assembled at different times and sorted by different opinions about what a good page looks like.

Four things have to happen, in order, for a general web search engine to work at all:

1. Crawling — a program fetches pages from the open web and follows the links out of them.
2. Indexing — the fetched pages are parsed, cleaned and stored in a structure that can be searched in milliseconds.
3. Ranking — when a query arrives, candidate pages are scored and put in an order.
4. Serving — the ordered results, plus whatever else the engine wants on the page, are assembled and sent to the browser.

An organisation that does all four owns a search engine. One that does only the last is a front end onto somebody else's — the ordinary case rather than the exception, since most search boxes on the web are wired into an index built by Google or Microsoft. Before any of that machinery matters, it is worth being precise about what a search engine is — and about which of the things people call search engines actually qualify.

Crawling: how pages get found in the first place

A crawler — also called a spider or a bot — is a program that requests a page over HTTP exactly as a browser would, stores what comes back, extracts the links and adds the new URLs to a queue. Repeat a few hundred billion times and you have the raw material for an index.

Real crawlers identify themselves. Google's crawls as Googlebot, in two main variants — Googlebot Smartphone, which does most of the work because Google indexes mobile-first, and Googlebot Desktop. Microsoft's is bingbot. Yandex documents roughly forty named robots, of which YandexBot is the main indexer. Others in daily use include Baiduspider, SeznamBot, Yeti for Naver in South Korea, MojeekBot, Bravebot, and the plainly-named search.marginalia.nu.

Crawlers are not supposed to take whatever they like. In 1994 Martijn Koster wrote the Robots Exclusion Standard — robots.txt — which lets a site state which paths a named crawler may fetch. It is a convention rather than a lock: it works because the major crawlers choose to honour it, and which of the newer AI crawlers honour it is among the sharpest disputes in search today.

Discovery is not purely passive. IndexNow, an open push-notification protocol launched in October 2021 by Microsoft and Yandex jointly and adopted by Seznam in August 2022, lets a site tell participating engines that a page has changed rather than waiting to be re-crawled. Brave supplements its crawl with the Web Discovery Project, an opt-in browser feature contributing anonymised records of which pages people actually visit.

Crawling is rate-limited, because it is somebody else's bandwidth being spent. Mojeek caps itself at one page request per second to any given site; Marginalia crawls from a small published IP range on an eight-to-ten-week refresh cycle, polling RSS feeds daily to fill the gap. Those constraints are exactly why a small engine's results are less fresh.

Indexing: turning pages into something searchable in milliseconds

A crawled page is a document; a query is a handful of words. Scanning every document for every query would be hopeless, so engines invert the problem and build an inverted index: rather than a list of pages each containing words, they store a list of words, each pointing at every page that contains it, with positions and weights attached. Retrieval becomes an intersection of a few word lists instead of a scan of the web.

Indexing is also where an engine decides what it will not keep. Google states plainly that it does not index every page it finds: low-value pages are crawled and discarded, duplicates are collapsed, and pages the crawler could not render or was told not to store drop out.

Index size is the honest measure of an engine's reach, and the figures that exist are worth knowing because they are so far apart:

• Mojeek reported 9 billion pages in 2025, up from 4 billion in June 2021.
• Qwant stated roughly 20 billion indexed pages in September 2023, with crawlers covering more than a billion pages a day.
• Marginalia reported around 300 million documents in 2024 — deliberately small, and chosen that way.
• Perplexity described an index tracking over 200 billion unique URLs in September 2025.

Neither Google nor Microsoft publishes a current index size; descriptions of Google's index in the hundreds of billions of pages are inferences rather than disclosures. The ordering, though, is not in doubt: Google's index is one to two orders of magnitude larger than any independent Western alternative, and that gap — not ranking cleverness — is what a reader notices first on switching engines and searching for something obscure.

Ranking: the part where engines genuinely differ

Crawling and indexing are engineering problems with broadly agreed solutions. Ranking is an argument about values, and it is where two engines holding similar pages produce visibly different results.

Ranking begins with relevance — does the page contain the query terms, in positions suggesting it is about the subject rather than merely mentioning it. Then come signals about the page and the site: how other pages link to it, how fresh it is, what language and region it serves, and hundreds of others that no major engine discloses in detail.

The famous one is PageRank, the link-counting method Larry Page and Sergey Brin published while at Stanford, which treats a link as a vote and weights each vote by the voting page's own score. It is worth being accurate about its status: PageRank was the idea that made Google better than AltaVista in 1998, it remains one signal among hundreds, and the public toolbar score was retired in 2016. "Google ranks by PageRank" has been wrong for roughly twenty years.

Alternatives have shipped. Teoma, out of Rutgers research and launched in 2001, ranked by Subject-Specific Popularity — counting link votes only from the cluster of pages that were authorities within the topic of the query, rather than from the whole web. Ask Jeeves bought it in September 2001 and ran it as ExpertRank until closing its own index in late 2010.

The most instructive modern counter-example is Marginalia, which inverts the usual value function on purpose: it downranks pages heavy with advertising and script, prefers text-dense documents, and boosts older pages. Its stated philosophy — that an index of a million high-quality documents beats an index of a billion where only a fraction are interesting — is not how Google works, and that is the point. Ranking is a choice, and different choices surface different webs.

One further difference matters to anyone comparing results: personalisation. Google adjusts results using signals about the person searching; Mojeek, Marginalia and DuckDuckGo do not, so everyone issuing the same query in the same region sees essentially the same page. That makes those engines reproducible, which matters whenever you need to show somebody else exactly what you saw.

The results page: what else is on it besides links

A modern results page is only partly a ranked list. Around and above the links sit paid listings, sold by auction and required to be labelled; knowledge panels and instant answers assembled from structured data; maps, images, video, news and shopping modules; and, increasingly, a generated summary at the top.

Some of that furniture is an engine's own work rather than a re-ranking of web pages. DuckDuckGo's Instant Answers are built from its own indexes over hundreds of specialist sources. Wolfram Alpha goes further and does not search the web at all — it computes answers from curated data and algorithms, which is why it can give you the distance to Neptune today but cannot find you a blog post.

Since 2023 the largest change to the results page has been the generated answer. Google launched AI Overviews as the Search Generative Experience in May 2023, brought them to all US users in May 2024 and to more than 100 countries on 28 October 2024; Gemini 3 became the default model behind them globally on 27 January 2026. Bing has carried a Copilot answer layer since February 2023. These summaries are grounded on the engine's existing index rather than on a separate crawl — the mechanics underneath are unchanged, and what changed is how much of the page goes to links.

Who pays for it, and what that money buys

Running a crawler and an index costs money continuously, so every engine has a funding model, and the model shapes the product.

Advertising is the dominant model, and it has a precise origin. In February 1998 Bill Gross's GoTo.com launched an open auction in which advertisers bid for placement against a keyword and paid only when someone clicked — a mechanism he presented at TED8 on 21 February 1998 to an audience that was reportedly baffled. GoTo became Overture, patented the method as US 6,269,361, sued Google over AdWords Select, and was bought by Yahoo in 2003. Every search ad since is a descendant.

Subscription is the main alternative. Kagi sells search by the month and takes no advertising, on the argument that with no advertiser the only party the ranking must satisfy is the person paying. The trade-off is that paying creates a billing identity a free engine never collects.

Selling index access has become a real business now that AI products need grounding data. Brave launched a search API in May 2023, Mojeek sells metered access to its index, and Perplexity began selling access to its own in September 2025. That market grew sharply after Microsoft retired all public Bing Search APIs on 11 August 2025, closing the cheap route by which small products had been reselling Bing.

Donations and grants sustain the smallest end. Marginalia reports operating costs of roughly $200 a month — a figure worth holding onto, because while it does not make Marginalia a rival to Google, it does show that the cost floor for an independent index of a few hundred million documents is about the price of a phone contract.

Why so few organisations actually do all of this

Building a search engine is not one hard problem but four, and the crawl never ends — the web changes underneath you, so an index is a thing you maintain rather than a thing you finish. The number of organisations crawling and ranking the open web at scale is consequently small enough to list.

Google and Microsoft in the United States; Yandex in Russia; Baidu and, in hybrid form, Sogou in China; Naver in South Korea; Seznam in the Czech Republic; Cốc Cốc in Vietnam. Independently in the West: Brave, which removed the last Bing API calls from its web results on 27 April 2023; Mojeek, a small British company that has never resold anyone's results; and Marginalia, which is one engineer and a server. Newer still: Perplexity and OpenAI, which built retrieval infrastructure for AI answers rather than for a results page, and the Ecosia–Qwant joint venture whose index Staan went live in August 2025.

Everything else with a search box — and there are hundreds — draws on one of those indexes under a commercial agreement. That is not a scandal, and such products are often perfectly good, but it determines what you are actually choosing when you change your default engine. Whether an engine crawls for itself, blends its own crawl with a partner's, or simply re-serves somebody else's results is the most useful and least advertised fact about it.

It can be checked. An engine that runs a crawler publishes it: a named user-agent, a way to verify the bot by reverse DNS or a published IP list, and instructions for blocking it — Mojeek's is at mojeek.com/bot.html, Google's in its developer documentation. An engine with no such page almost certainly has no crawler.

Frequently asked questions

Sources

• developers.google.com/search/docs/crawling-indexing/google-common-crawlers
• blogs.bing.com/webmaster/april-2022/Announcing-user-agent-change-for-Bing-cr…
• mojeek.com/bot.html
• about.marginalia-search.com/article/crawler/
• indexnow.org/
• brave.com/blog/search-independence/
• learn.microsoft.com/en-us/lifecycle/announcements/bing-search-api-retirement
• research.perplexity.ai/articles/architecting-and-evaluating-an-ai-first-sear…