How Search Engine Indexing Works

By Bill Hartzer · Updated 19 August 2026

The index is the web turned inside out

When a search engine returns results in under a second, it is not looking at the web. It is looking at a structure it built earlier from a crawl, and the shape of that structure is the reason the operation is fast.

A crawled page is a document that contains words. That is the wrong way round for searching. Scanning several hundred billion documents for a word, on demand, is not tractable at any price. So the indexer inverts the relationship and stores, for each distinct word, the list of documents containing it. This is the inverted index, and it has been the core data structure of information retrieval since long before the web. Ask it for marginalia and it hands back a posting list of document identifiers immediately, because that list is what it stores.

Intersecting two such lists gives every document containing both words. Storing the position of each word within each document — which every serious engine does — additionally allows phrase queries, because you can check whether the positions are adjacent. Almost everything a search box does, including the quoted-phrase and boolean operators that survive from the 1990s, falls out of that one structure.

The engine that first proved this worked at web scale was AltaVista, launched in December 1995 out of Digital Equipment Corporation's research labs. Its predecessors indexed titles, filenames or human-written descriptions; AltaVista indexed the full text of every page it fetched and held the index in memory. The indexer was built substantially by Michael Burrows, later a co-author of the Burrows–Wheeler transform. That combination — a fast multi-threaded crawler feeding a RAM-resident full-text index — is the moment indexing the whole web stopped being theoretical.

From fetched bytes to index entry

Between the crawler's response and the posting list sits a processing pipeline. The steps are unglamorous and every engine performs some version of them.

Parsing. The HTML is turned into a document tree, the boilerplate is separated from the content, and the visible text is extracted along with structural signals — what is a heading, what is a list, what is a link and where it points. Engines that render JavaScript, such as Google and Baidu with its dedicated Baiduspider-render agent, run a browser engine at this stage; those that do not, index whatever the server sent.

Tokenising. The text is split into terms. This is trivial in English and genuinely hard elsewhere: Chinese and Japanese are written without spaces between words, which is a significant part of why Baidu and Sogou built their own tokenisers rather than adapting Western ones, and why Cốc Cốc's handling of Vietnamese diacritics is treated as engineering rather than configuration.

Normalising. Case is folded, accents are handled, and morphological variants are collapsed so that a search for one form matches another. Doing this well is language-specific, which is one reason engines have uneven quality across languages — a crawler is language-agnostic, an indexer is not.

Language and encoding detection. The engine records what language the document is in, because ranking and query interpretation both depend on it. Cốc Cốc's crawler explicitly limits itself to Vietnamese content, so this determination is not a refinement for it but a filter.

Signal extraction. Alongside the words, the indexer stores metadata: the title, the outbound links and where they point, the anchor text other pages use when linking in, structured data markup, publication dates where they can be determined, and internal quality scores. Anchor text is historically important — it lets an engine index a document by words that do not appear anywhere in it, using the descriptions other people wrote.

Crawled is not indexed

The most persistent misconception about indexing is that it is automatic — that anything a crawler fetches ends up searchable. It does not. Selection happens, and it happens on a large scale.

Google states openly that it does not index every page it finds and that low-value pages are crawled and then discarded. Every engine does this, because storage and serving cost money per document, forever, and a large fraction of what a crawl returns is duplicate, machine-generated, empty, spam, or an infinite variation on a URL parameter. An index is a curated subset by necessity, and the curation is a ranking decision made before any query exists.

Marginalia makes its selection criteria unusually explicit, and inverts the usual ones. Heavy JavaScript, dense advertising, affiliate links and tracking scripts all push a document down; age is treated as a positive signal under a stated Lindy-effect assumption that content which has survived is more likely to be worth reading. Kagi's Teclis index is likewise built by deliberately focusing its crawl on "non-commercial, high-quality content", and Kagi does not claim it is a general index of the whole web. These are editorial positions embedded in an index, and they are the reason the same query returns genuinely different worlds on different engines.

It is worth separating two controls that are constantly confused at this point. A robots.txt rule governs crawling — whether the fetch happens at all. A noindex directive, delivered in a meta tag or an HTTP header, governs indexing — whether a fetched page is stored and served. They operate at different stages and one cannot substitute for the other.

Duplicates, canonicals and near-identical documents

A very large share of the raw web is duplicate. The same article appears on a print URL, a mobile URL, a syndicated partner, an AMP variant and half a dozen tracking-parameter permutations, and a naive index would store all of them and return all of them.

So indexers cluster documents they judge to be the same or near-same, pick one representative — the canonical — and consolidate the signals from the rest onto it. The alternatives usually stay in the index in some reduced form so the engine can still resolve a direct request for them, but only the canonical competes in results.

Doing this cheaply at scale is a hashing and fingerprinting problem rather than a comparison problem: engines compute compact signatures of document content that can be compared without re-reading the documents. Getting it wrong in either direction is costly — collapse too aggressively and distinct pages disappear; collapse too little and results fill with the same text repeated.

Sharding, serving and the physical cost

No single machine holds a web index. It is split into shards across a large number of servers, each holding a slice of the documents; a query is broadcast to all shards, each returns its own best candidates, and the results are merged. That design is why adding documents costs hardware roughly linearly, and why index size is a capital expenditure rather than a software achievement.

The scale is worth stating concretely, because it explains the shape of the market. Perplexity's September 2025 engineering write-up describes "an exabyte-scale index and crawling apparatus" covering over 200 billion unique URLs, running on "tens of thousands of CPUs and hundreds of terabytes of RAM", with tens of thousands of indexing operations per second to maintain freshness. At the other end, Marginalia's creator has run a real web index on hardware that started as a PC on domestic broadband and migrated to a single server in October 2023, with crawl data given a stated shelf life of two to three months.

Both are genuine indexes. The distance between them is the whole argument about whether independent search is viable, and Marginalia is the strongest available evidence that it is — while Perplexity's figures are the strongest available evidence of what matching an incumbent actually costs.

How big an index actually is

Index size is the number everyone wants and almost nobody publishes reliably. The figures that do exist are self-reported, dated, and not measured the same way.

Mojeek publishes a milestone history: one billion pages in 2015, 2.3 billion in May 2019, 4 billion in June 2021, 6 billion in October 2022, 8 billion in 2024 and 9 billion as of 2025.
Qwant self-reported around 20 billion pages in September 2023.
Perplexity claims coverage of "hundreds of billions of webpages" and over 200 billion unique URLs as of September 2025.
Seznam published 30 million documents at its 2005 launch and a rise from 400 million to 700 million by 2012, but has published no current figure.
Google does not publish an index size at all. It is generally understood to be in the hundreds of billions of pages.

Two cautions apply to all of these. First, a URL count and a stored-document count are not the same thing, and operators are not consistent about which they quote. Second, size is a poor proxy for usefulness — an index of nine billion carefully selected documents can answer a research question that an index of two hundred billion cannot, if the two hundred billion are weighted toward commercial pages. What size does determine is recall: whether the engine has the page at all. On that specific question, Mojeek at nine billion pages will simply not hold the document on a substantial share of long-tail queries, and that is the honest limit of a small independent index.

Indexing for retrieval versus indexing for an answer

The classic inverted index matches terms. That is a poor fit for a system whose output is a generated paragraph rather than a list of links, and the AI search products have adapted the structure accordingly.

Two changes stand out. The first is passage-level indexing: instead of treating a document as the unit of retrieval, the index stores and scores sub-document passages, so a system can pull the two relevant paragraphs out of a long page rather than handing an entire page to a language model. Perplexity describes retrieving and ranking at both document and sub-document level explicitly, and says the architecture was built for feeding a model rather than rendering a results page.

The second is embedding-based indexing, which stores documents as vectors capturing meaning rather than as term lists, so that a query can match a passage with no words in common with it. In practice the serious systems use both: a lexical index for precision and recall on exact terms, an embedding index for semantic matching, and a reranking stage over the combined candidate set. Perplexity's published pipeline is exactly this — lexical and embedding scorers for candidate generation, then cross-encoder rerankers.

The underlying economics have not changed. Whichever structure is used, someone has to crawl the documents, store them, keep them fresh and pay for the machines they sit on. Every AI search product is standing on an index, and the ones that do not own theirs are renting.

Frequently asked questions

What is a search engine index?

It is the engine's own processed, stored copy of the pages its crawler fetched, restructured so it can be queried instantly. The core structure is an inverted index: rather than storing which words are on each page, it stores which pages contain each word. Searching consults that structure, never the live web, which is why results return in milliseconds.

What is an inverted index?

An inverted index maps each distinct term to the list of documents containing it, plus the positions where it appears. Intersecting two of those lists gives every document containing both words, and comparing positions supports phrase searching. It is the standard data structure of information retrieval and predates the web by decades.

Does a search engine index every page it crawls?

No. Google states plainly that it does not index every page it finds and discards low-value ones. Selection is unavoidable because storage and serving cost money per document indefinitely, and much of what a crawl returns is duplicate, machine-generated or empty. Every index is a curated subset chosen before any query exists.

What is the difference between crawling and indexing?

Crawling is fetching a page over HTTP; indexing is processing and storing it so it can be found. They are separate stages with separate controls: a robots.txt rule blocks the fetch, while a noindex directive tells an engine not to store and serve a page it did fetch. Neither can do the other's job.

How big is Google's index?

Google does not publish a figure. It is generally understood to be in the hundreds of billions of pages, and Google has been explicit that it crawls considerably more than it keeps. For comparison, Mojeek reports around 9 billion pages as of 2025 and Perplexity claims over 200 billion unique URLs as of September 2025 — all self-reported and measured differently.

How do search engines handle duplicate pages?

They cluster documents judged to be the same or near-identical, choose one canonical representative and consolidate signals onto it, so only one competes in results. The comparison is done with compact content fingerprints rather than by re-reading documents. Getting it wrong in either direction is costly: over-collapsing hides distinct pages, under-collapsing floods results with the same text.

Why do different search engines return different results for the same query?

Partly ranking, but mostly the index. Two engines crawl different subsets of the web, apply different selection rules, and store different signals. Marginalia deliberately downranks commercial, script-heavy pages and treats age as a positive; Kagi's Teclis index is focused on non-commercial content. Different indexes cannot agree, whatever the ranking does.

What is passage indexing?

It is indexing at sub-document level, so an engine can retrieve the specific relevant paragraphs of a long page rather than the whole page. It matters most for AI answer systems, which need to hand a language model a small amount of relevant text. Perplexity documents retrieving and ranking at both document and passage level for exactly this reason.

How search engine indexing works