A
AI Overviews also: SGE, Search Generative Experience
Google's AI-generated summary shown above the conventional results for many queries, with links to the pages it drew on. It began as the Search Generative Experience in Search Labs in May 2023, reached all US users in May 2024, and expanded to more than 100 countries in October 2024. The summaries are produced by Gemini models grounded on Google's existing web index rather than by a separate crawl. AI Overviews matter because they change what a results page is for: the answer is on the page, and the cited publisher may never receive the visit. There is no documented account-level switch to turn them off; a per-search Web filter returns plain links instead.
Algorithm
The set of rules, models and scoring functions a search engine uses to decide which documents from its index answer a query, and in what order. Modern ranking is not one algorithm but a pipeline: candidate retrieval from an inverted index, scoring by relevance functions such as BM25, then re-ranking by machine-learned models that weigh link authority, freshness, language match and user-interaction data. It matters because the algorithm, not the index, is what makes two engines holding similar documents return different pages. Engines disclose very little of it: Google describes families of systems rather than weights, Naver names its C-Rank and D.I.A. systems without publishing them, and the rare exception was the January 2023 leak of Yandex source code, which analysts reported contained a long list of ranking factors.
Answer engine also: Question-answering system
A search product whose output is a written answer rather than a ranked list of links. Perplexity is the clearest example: a natural-language question returns prose with numbered inline citations to the sources retrieved, plus follow-up questions. The category is older than the current AI wave — Ask Jeeves took natural-language questions from 1996, and Wolfram Alpha, launched 18 May 2009, computes answers from curated structured data without touching the web at all. What distinguishes today's answer engines is that a large language model writes the response from documents fetched at query time. The trade-off is structural: an answer is faster to read and harder to verify than ten links, and the citation is the only bridge back to the source.
Autocomplete also: Query suggestions, Search suggestions, Type-ahead
The list of predicted queries an engine offers as you type into the search box. Predictions are generated from aggregated past queries, trending searches and, on some engines, the language of the index itself; they are commonly filtered by policy rules that suppress suggestions in sensitive categories. Autocomplete matters more than its convenience suggests. It shapes what people actually search for, so it partly determines its own training data, and because suggestions are visible and reproducible they are a recurring subject of complaint, litigation and regulatory attention in several countries. Engines that do not personalise, such as Mojeek, will show every user the same suggestions for the same prefix; engines that build user profiles need not.
B
Bang commands also: Bangs, !bang
A query prefix that routes a search to another site instead of the engine's own results. DuckDuckGo popularised the syntax — typing !w einstein jumps straight to the Wikipedia article, !a to Amazon — and maintains thousands of them. SearXNG uses the same idea to force a single upstream engine, so !go asks Google and !wp asks Wikipedia. Bangs matter because they make a general engine into a launcher for the specialised search boxes that a web crawler covers poorly, such as a library catalogue or a code repository. They also reveal something structural: an engine willing to send a query away immediately is competing on being the default box, not on holding every answer.
BERT also: Bidirectional Encoder Representations from Transformers
A transformer-based language model published by Google researchers in 2018 and applied to Google Search from October 2019. Unlike earlier models that read text left to right, BERT reads a sequence in both directions at once, so the meaning it assigns to a word depends on the words on either side of it. Google's stated benefit was better handling of prepositions and word order in conversational queries, where dropping a small word changes the question entirely. BERT matters historically because it marks the point where a general-purpose pretrained language model, rather than hand-built query rules, became core ranking infrastructure — the direct ancestor of the large language models now generating answers on results pages.
BM25 also: Okapi BM25, Best Match 25
A probabilistic ranking function that scores how well a document matches a query, developed by Stephen Robertson, Karen Sparck Jones and colleagues on the Okapi system at City University London. It refines TF-IDF in two ways: repeated occurrences of a term give diminishing returns rather than scaling linearly, and scores are normalised for document length so a long page cannot win simply by containing more words. BM25 matters because it is the default lexical baseline of the whole field — it is what open-source engines such as Lucene, Elasticsearch and OpenSearch use out of the box, and what newer neural retrieval methods are measured against. Most production systems now blend a BM25-style score with embedding-based retrieval rather than replacing one with the other.
Boolean operators also: AND OR NOT, Boolean search
The logical connectives AND, OR and NOT used to combine query terms, so that a search returns documents matching a stated logical condition rather than a fuzzy relevance guess. Boolean retrieval was the dominant model in early web search: AltaVista offered weighted Boolean queries with field and date restrictions, Gigablast kept a rich Boolean and phrase syntax to the end, and Magellan accepted + and - prefixes. It matters because Boolean queries are exact and reproducible — the same query returns the same set — which is why professional legal, patent and library databases still use them. Mainstream web engines have steadily narrowed their support, treating operators as hints to a ranking model rather than strict conditions.
C
Cached page also: Cache, Search engine cache
A copy of a page as the crawler last saw it, stored by the engine and, historically, shown to users through a cached link beside the result. The cache exists first as working infrastructure — ranking and snippet generation run against stored copies, not live fetches — and only incidentally as a user-facing feature. It mattered as a public record: a cached copy let anyone read a page that had gone offline, changed or been quietly edited, which made it a standard tool for researchers and journalists. Google removed the cached link from its results in early 2024, saying it had been added when pages often failed to load and was no longer needed. Web archives, not engines, now fill that role.
Canonicalisation also: Canonical URL, rel=canonical, Canonical tag
The process by which a search engine decides which of several URLs holding the same or near-identical content is the one to keep in its index. The same page is routinely reachable at many addresses — with and without www, over HTTP and HTTPS, with tracking parameters attached, or through printer-friendly variants — and an engine that stored them all would waste index space and split the evidence about the page across duplicates. Engines pick a canonical using signals including the rel="canonical" link element, redirects, internal links and sitemap entries; the declaration is a hint, not a command, and engines regularly choose differently. Canonicalisation is the step that turns a set of crawled URLs into a set of distinct documents.
Choice screen also: Search engine ballot, Preference menu
A screen presented during device or browser setup that asks the user to pick a default search engine from a list, instead of one being set for them. Choice screens exist because regulators ordered them: Russia's competition authority required one on Android in 2017 following Yandex's complaint against Google, the European Commission required one after its 2018 Android decision, and the EU Digital Markets Act — under which Alphabet was designated a gatekeeper on 6 September 2023 — imposes obligations of this kind on designated services. They matter because default placement, not product quality, decides most search market share. Smaller engines treat inclusion as a distribution channel; Presearch, for instance, secured a place on the EU Android choice screen.
Citation also: Source link, Attribution
In AI search, the link an answer engine attaches to a generated sentence to show which retrieved document it came from. Perplexity made numbered inline citations its defining interface from launch in December 2022, and Google's AI Overviews, Microsoft Copilot and ChatGPT Search all adopted some form of the pattern. Citations matter because a generated answer is otherwise unfalsifiable: the link is the only route from a claim back to evidence. They are also weaker than they appear. A citation records that a document was retrieved, not that the sentence attached to it reflects that document — reporting by Forbes and Wired in June 2024, and a News Corp lawsuit filed the same year, alleged answers that misattributed or fabricated material.
Click-through data also: Click logs, Interaction data, Query logs
The record of which results users clicked for which queries, and what they did next. It is one of the most valuable assets a large engine owns, because it is a continuous stream of human relevance judgements at a scale no editorial process could match, and it compounds: the engine with the most users gets the most feedback and improves fastest. That feedback loop was central to the US antitrust case against Google — the September 2025 remedies decision ordered Google to share certain click-and-query data with qualified competitors, alongside five-year syndication licences. Engines that decline to build user profiles, such as Mojeek, forgo this signal entirely and rank from document and link evidence alone.
Crawl frontier also: URL frontier, Crawl queue
The prioritised queue of URLs a crawler has discovered but not yet fetched. Because the web is effectively unbounded and any crawl has a finite budget of bandwidth and machines, the frontier is where an engine decides what the web is for its purposes: which hosts to revisit and how often, which newly discovered links to follow, which to defer indefinitely. A frontier scheduler must also respect per-host politeness limits, avoid crawler traps such as infinitely generated calendar pages, and balance breadth against re-crawling pages that change. The frontier explains why two engines starting from the same seed URLs end up with different indexes, and why a page can be known to an engine for a long time without ever being fetched.
Crawler also: Spider, Bot, Robot, Web crawler
The automated program that fetches pages so an engine can index them. A crawler starts from a set of seed URLs, downloads each page, extracts its links, adds them to the crawl frontier, and repeats — identifying itself in a User-Agent string and consulting robots.txt before it fetches. Running one at web scale is the single hardest and most expensive thing about operating a search engine, which is why so few organisations do it: Googlebot, Microsoft's Bingbot, MojeekBot, Bravebot, Baidu's Baiduspider, Naver's Yeti and PerplexityBot are most of the meaningful list. An engine with no crawler of its own — Startpage, Dogpile, SearXNG — is reselling or aggregating someone else's crawl.
Crawling also: Discovery, Fetching
The first stage of web search: discovering URLs and downloading their content. Engines find pages mainly by following links from pages they already hold, supplemented by sitemap files, feeds and submission interfaces. Crawling is distinct from indexing — a page can be fetched and then discarded as duplicate, empty or unreachable, and a page blocked from crawling can still appear in results if the engine has learned about it from links elsewhere. Its economics shape the industry: the crawl must run continuously to stay current, costs scale with the size of the web rather than with the number of users, and that fixed cost is the main reason most self-described alternative search engines buy results instead of collecting their own.
D
Deduplication also: Near-duplicate detection
Identifying documents that are the same or nearly the same and keeping only one in the index or the results. A large share of the crawled web is duplicated — syndicated articles, mirrored documentation, product pages generated from one template, and the same page served at many URLs — and an engine that returned all copies would fill a results page with one document. Techniques range from exact checksums to shingling and similarity hashes such as SimHash and MinHash, which catch near-duplicates differing only in boilerplate. Deduplication matters to meta-search engines for a second reason: Dogpile, MetaCrawler and ProFusion merge feeds from several engines whose results overlap heavily, so removing repeats is most of what merging means.
Deep web also: Invisible web, Hidden web
The parts of the web that general crawlers do not reach: content behind logins, paywalls and forms, database records generated only in response to a query, and material excluded by robots.txt. It is not the same as the dark web, which means content on overlay networks such as Tor, and it is far larger than the crawlable surface web. The deep web matters because it holds a great deal of the material people most need — court records, patents, academic databases, library catalogues. Gateways to it were a real product category: after Intelliseek bought it in 2000, ProFusion was relaunched as an Invisible Web gateway over a claimed 1,000-plus specialist sources, with a companion directory listing thousands more.
Default search engine also: Default engine, Search default
The engine a browser or device uses when a query is typed into its address bar or search box, without the user choosing one per search. Defaults decide market share, because most people never change them: this is why search engines pay browser and handset makers for the position, and why those payments are large enough to be the centre of competition cases on two continents. The US court's September 2025 remedies in the Google search case barred exclusive default agreements while stopping short of banning payments for placement. Changing the default is a per-application setting, not a system-wide one, which is why a user who switches in one browser often finds another still routing to Google.
Directory also: Web directory, Human-edited directory
A catalogue of websites arranged by subject and compiled by people rather than by a crawler. Directories were the dominant way to find things in the mid-1990s: Yahoo! began in 1994 as a hand-built hierarchy, LookSmart and Magellan sold or reviewed listings, and the Open Directory Project (DMOZ), launched 5 June 1998, was maintained by volunteer editors and licensed its data out to other engines until it closed on 17 March 2017. A directory lists sites with human descriptions; a search engine indexes pages and matches text. The model failed on scale and freshness — human editing could not keep pace with the growth of the web — but it produced curation and consistency that automated ranking has never fully replaced.
E
Embedding also: Vector, Vector search, Dense retrieval
A representation of a word, passage or query as a list of numbers positioned in a high-dimensional space, arranged so that items with similar meanings sit close together. Embeddings let an engine retrieve documents by proximity in that space rather than by shared words, so a query about how much a puppy costs can match a page that says price of a dog. This is what makes semantic search practical, and it underpins the retrieval stage of AI answer engines. It has a characteristic failure: because matching is by similarity rather than by presence of terms, results can be topically close but literally wrong. Production systems generally combine embedding retrieval with lexical scoring — Perplexity describes a multi-stage lexical, embedding and cross-encoder pipeline.
Entity also: Named entity
A distinct thing an engine can identify and hold facts about — a person, place, company, product, film or concept — as opposed to a string of characters. Recognising that jaguar may denote an animal, a car marque or an American football team, and choosing between them from context, is entity resolution, and it is what lets an engine answer about a thing rather than merely match words on a page. Entities are stored in a knowledge graph with typed relationships between them, which is what powers knowledge panels and direct answers. Yahoo cites a knowledge graph of roughly one billion entities behind its Scout answer product, launched in beta on 27 January 2026; those are the company's own figures, not audited ones.
F
Featured snippet also: Answer box, Position zero
A block at the top of a results page that quotes an extract from one ranking page as a direct answer, usually as a paragraph, list or table, with a link to the source. Featured snippets predate generative AI and work differently: the text is selected and copied from an existing document rather than written by a model, so it can be traced to a single page. They matter because they are the origin of the zero-click pattern — a user who reads the extract has no reason to open the source — and because they are also what voice assistants read aloud when asked a question. AI Overviews have displaced them on many queries, synthesising several sources instead of quoting one.
Federated search also: Distributed search, Broadcast search
Searching several independent collections at once by sending the query to each and merging what returns, instead of building one combined index. The distinction from crawling is that the documents stay where they are and are searched in place, which is why federated search is standard in libraries and enterprises, where content sits in catalogues, subscription databases and internal systems that cannot be centrally indexed. Web meta-search engines are the consumer form of the same architecture. Its hard problem is merging: the constituent systems return incomparable relevance scores, so the merged ranking is an estimate. ProFusion, built at the University of Kansas in 1996, addressed this by learning how far to trust each underlying engine for each kind of query.
Filter bubble also: Personalisation bubble
The narrowing of what a person sees when an engine tailors results to their inferred interests, so that material confirming existing preferences is favoured and dissenting material is less likely to surface. The term was popularised by Eli Pariser in 2011. Empirical work since has been mixed — several studies find web results less personalised than assumed, with location and language accounting for much of the variation people notice — so it is better treated as a live question than a settled fact. It matters structurally because it separates two designs: engines that log and profile, and engines such as Mojeek that return identical results to every user for the same query. The second is reproducible, which is valuable for research and comparison.
Freshness also: Recency
How current an index is, and how strongly recency is weighted at ranking time. Freshness is a scheduling problem before it is a ranking one: an engine must decide how often to re-crawl each page, since a news homepage changes hourly while an archived document may never change, and re-crawling everything constantly is unaffordable. It matters because for a large class of queries a stale answer is a wrong answer, and because it is one of the clearest points of separation between large and small indexes — a small independent crawler can cover a great deal of the web but will lag on how fast it notices changes. Brave supplements its crawl with an opt-in browser telemetry system to learn which URLs are worth revisiting.
G
Grounding also: Retrieval grounding, Web grounding
Constraining a language model's answer to documents retrieved at query time, rather than letting it answer from what it absorbed during training. In a grounded system the query triggers a search, the retrieved passages are placed in the model's context, and the answer is generated from them with citations back to the sources. Grounding matters because it is the main defence against hallucination and the only way an AI answer can reflect events after the model's training cut-off. It also explains the market for search APIs: an AI product needs an index, and few companies own one. Google grounds AI Overviews on its own index; Yahoo's Scout uses Microsoft's Bing grounding API; Microsoft sells Grounding with Bing Search through Azure.
H
Hallucination also: Confabulation
A fluent, confident statement produced by a language model that is not supported by any source — an invented citation, a misattributed quotation, a plausible but non-existent fact. Hallucination is a property of how these models work rather than a bug awaiting a patch: they generate text that is probable given the context, and probability is not truth. It matters acutely in search because the traditional failure mode of a results page was a bad link the user could see was bad, whereas a hallucinated answer looks exactly like a correct one. Grounding and citation reduce the rate but do not eliminate it; a News Corp suit against Perplexity in 2024 alleged fabricated quotations attributed to its publications.
Hubs and authorities also: HITS, Link analysis
A link-analysis model, published by Jon Kleinberg in 1999 as the HITS algorithm, that scores pages in two roles at once: an authority is a page many good hubs link to, and a hub is a page that links to many good authorities. Unlike PageRank, which computes one global score for the whole web, HITS is computed over the subgraph relevant to a particular query. It matters because it is the intellectual root of topic-restricted authority. Teoma, built from Rutgers University research and launched in 2002, ranked using what it called Subject-Specific Popularity — counting link votes only from within the query's own topical community — and its Refine and Resources panes were direct byproducts of that clustering.
I
Index also: Search index
The engine's own stored, searchable copy of the web — the data structure built from crawled pages that a query is actually run against. A search does not touch live websites; it touches the index, which is why results can lag reality, and why an engine's coverage is bounded by what its crawler has fetched and kept. Owning an index is the dividing line in this industry. Google, Bing, Yandex, Baidu, Mojeek, Brave Search and a short list of others build their own; Startpage, Yahoo! and Dogpile hold none and serve results licensed from those that do. The distinction determines whether an engine can differ from its supplier at all.
Index size also: Index coverage, Number of indexed pages
The number of documents an engine holds. Figures are hard to compare and easy to misread: engines count different things, publish rarely, and have every incentive to flatter themselves, so most stated sizes are self-reported and undated. The published ones still show the scale of the gap. Mojeek reported passing one billion pages in 2015, four billion in June 2021 and around nine billion by 2025; Qwant self-reported roughly twenty billion in September 2023; Perplexity claimed an index tracking over 200 billion unique URLs in September 2025; Google's is generally understood to run to hundreds of billions. Size is not quality — coverage of the long tail, freshness and ranking all matter — but below a certain scale an engine simply cannot answer rare queries.
Indexing also: Index building
The stage that turns a fetched page into something searchable: the HTML is parsed, JavaScript may be executed, text is extracted and tokenised, duplicates are resolved, links are recorded, language is detected, and the resulting terms are written into the inverted index along with metadata used at ranking time. Indexing is where most of an engine's editorial judgement lives, because it is where pages are dropped — as duplicate, spam, empty after rendering, or simply not worth the storage. The distinction from crawling matters and is widely muddled: crawling is fetching, indexing is keeping. A robots.txt rule governs the first; a noindex directive governs the second, and only works if the page is crawled.
Intent also: Search intent, Query intent
What the person actually wants from a query, as distinct from the words they typed. The conventional taxonomy comes from Andrei Broder's 2002 paper: navigational queries seek a particular site, informational queries seek knowledge, and transactional queries seek to do something. Intent matters because it determines the shape of the answer, not just its content — the same three-word query may deserve a map, a definition, a set of product listings or a conversation. Modern engines infer it from the query text, from what other people who typed it went on to click, and from context such as location and device, which is why one query can return a local pack in one country and a knowledge panel in another.
Inverted index also: Postings list, Inverted file
The core data structure of text search: instead of storing, for each document, the words it contains, it stores for each word the list of documents containing it. That list is a postings list, and it usually records positions too, so phrases can be matched. It is called inverted because it reverses the natural arrangement, and it is what makes web search possible at all — answering a two-word query means intersecting two precomputed lists rather than reading billions of pages. Every serious engine is built on one, from AltaVista in 1995 to Lucene-based systems today. Its constraints explain much of search behaviour: terms are normalised before they are stored, so what is retrievable is decided at indexing time, not query time.
J
JSON-LD also: JSON for Linking Data
A syntax for embedding structured data in a page as a self-contained <script type="application/ld+json"> block, rather than by annotating HTML elements inline as microdata and RDFa do. It is a W3C recommendation and the format search engines most commonly document for schema.org vocabulary. JSON-LD matters because it separates the machine-readable description of a page from its markup: the data can be generated and validated independently of layout, and the same block can describe an article, its author and its publisher as linked objects. Engines parse it to identify entities and to decide whether a result qualifies for a rich result; parsing it is not the same as believing it, and unsupported claims in markup are routinely ignored.
K
Knowledge graph also: Entity graph
A database of entities and the typed relationships between them — person, born in, place — used by a search engine to answer about things rather than to match strings. Google announced its Knowledge Graph in May 2012, drawing on sources including Freebase and Wikipedia; Yandex, Bing, Baidu and Naver run equivalents, and Yahoo cites a roughly one-billion-entity graph behind its Scout answer product. Knowledge graphs matter because they turn search from document retrieval into fact retrieval: they power knowledge panels, direct answers and the disambiguation that decides which Cambridge a query means. They also concentrate authority — when an engine states a fact from its graph rather than linking to a page, there is no source on screen to check.
Knowledge panel also: Info box, Entity panel
The boxed summary of an entity that appears beside or above results — a person's dates and occupation, a company's founding and headquarters, a film's cast — assembled from an engine's knowledge graph and from licensed or open sources rather than written for the query. Panels matter because they are the most visible form of zero-click search: the fact is on the results page, and often no click follows. They also expose how much of an engine's page is its own content. Not every engine has one built from its own data; Qwant draws knowledge panels from Wikipedia and Wikimedia, and Startpage added panels through a partner integration in March 2023 rather than building an entity store.
L
Large language model also: LLM
A neural network trained on very large volumes of text to predict continuations, which in practice lets it summarise, answer and converse. In search, LLMs occupy two distinct roles that are easily confused: understanding the query, and writing the answer. The first is older and quieter — BERT and RankBrain are ranking infrastructure, invisible to the user. The second is what changed the results page after 2022, when generated prose began appearing above the links. Models do not hold a current index; without grounding they answer from training data with a fixed cut-off and no way to distinguish recall from invention. Google uses Gemini for AI Overviews, and Yahoo's Scout is built on Anthropic's Claude with Bing grounding.
Link graph also: Web graph, Backlink graph
The web modelled as a network: pages are nodes and hyperlinks are directed edges. Treating links as evidence — a link is a deliberate act by one author pointing at another's work — is the insight that separated second-generation engines from first. PageRank and HITS are both computations over this graph, and the graph is also how a crawler discovers the web at all, since every unfetched URL was found on a fetched page. It matters because building one requires a crawl: an engine that resells another's results has no link graph of its own, and cannot rank differently on that basis. It is also the most heavily manipulated structure in search, which is why raw counts have long been discounted.
Literal search also: Verbatim search, Exact-phrase search, Exact match
Searching for exactly the characters entered, with none of the transformations engines normally apply — no stemming, no synonym expansion, no dropping of stop words, no spelling correction, no substitution of a related query the engine thinks you meant. On most engines the request is made with quotation marks around a phrase, and Google additionally offers a Verbatim setting under its search tools that suppresses these rewrites for the whole query. Literal search matters when the string itself is the object of interest: an error message, a line of code, a legal phrase, a name that doubles as a common word, or checking whether a wording appears on the web at all. Engines vary in how strictly they honour it, and rarely document the limits.
Local pack also: Map pack, Local results
The map-backed block of nearby businesses shown for queries an engine reads as having local intent, typically three listings with address, hours and rating above the ordinary web results. It is drawn from a business-listing database and geographic data rather than from the web index, which is why the entries are records rather than pages. Local results matter because they answer a large share of mobile queries and because the underlying database is owned by the engine, making it a clear case of an engine's own content occupying the top of its own page. Coverage varies sharply by market: an engine without a local dataset either licenses one or leaves the query to a mapping service.
M
Meta-search engine also: Metasearch, Search aggregator
An engine with no crawler and no index that answers a query by sending it to other engines and merging what comes back. The model was central to the 1990s web — MetaCrawler (1995), SavvySearch, Dogpile, Mamma, ProFusion and Highway 61 all worked this way — and survives in SearXNG, which supports up to 269 upstream services and can be self-hosted. Meta-search matters for two reasons: it was a rational answer to a fragmented web where no single index was complete, and it is the architecture behind most privacy front-ends, since the upstream engine sees the aggregator's server rather than the user. Its weakness is total dependence — when one engine became clearly best, merging had little left to add.
Mobile-first indexing
Indexing and ranking a site from the version its crawler is served on a mobile user-agent, rather than from the desktop version. Google announced the shift in 2016 and rolled it out over several years as most searches moved to phones; the practical consequence is that the mobile rendering of a page, not the desktop one, is what the index holds. It matters historically because it inverted a long-standing assumption that desktop was canonical and mobile a reduced copy, and because it made the crawler's own user-agent identity load-bearing — Googlebot Smartphone now does most of Google's crawling. Other engines have made the same move at different times and with less documentation, so behaviour is not uniform across the industry.
N
Natural language processing also: NLP, Computational linguistics
The field concerned with getting computers to work with human language — segmenting text into words, resolving what a pronoun refers to, recognising entities, parsing a question's structure, translating. Search has driven NLP and been reshaped by it in turn: early engines matched strings and left meaning to the user, while every modern stage from tokenisation and stemming through query understanding to generated answers is an NLP problem. The line from Google's Hummingbird update in 2013, through BERT in 2019, to today's large language models is one continuous shift from matching words to modelling meaning. Non-English search depends on it more heavily, since languages such as Chinese and Japanese do not mark word boundaries with spaces.
Noindex also: noindex directive, Robots meta tag
An instruction, given as a <meta name="robots" content="noindex"> tag or an X-Robots-Tag HTTP header, telling search engines not to keep a page in their index. It is the counterpart to robots.txt and the pair are constantly confused: robots.txt controls whether a crawler may fetch a page, noindex controls whether the engine may store it. The consequence is counter-intuitive — a page blocked in robots.txt cannot be read, so its noindex is never seen, and the URL may still be listed on the strength of links pointing at it. Compliance is voluntary in both cases; it rests on convention, not enforcement, and crawlers that ignore both exist.
O
Organic results also: Natural results, Algorithmic results
The unpaid listings an engine returns because its ranking judged them relevant, as opposed to advertisements, which appear because someone bid on the query. The distinction is the foundational bargain of commercial search, and it depends on the two being visually separable — regulators in several jurisdictions require paid placements to be labelled for exactly this reason. The term has grown less precise as results pages have filled with material that is neither: knowledge panels, local packs, shopping units, an engine's own properties and AI Overviews are not ads, but they are not ranked web documents either. On many commercial queries the first organic link now sits well below the fold, which is a structural change rather than a cosmetic one.
Own index also: Independent index, Index ownership
Whether an engine crawls and ranks the web itself, or serves someone else's results. It is the most useful and least-known fact about any search engine, because an engine without an index cannot return anything its supplier did not. Only a handful of organisations run a general web index — Google, Microsoft, Yandex, Baidu, and independents such as Mojeek, Brave Search and Perplexity. Many well-known names do not: Yahoo! has been served by Bing since 2010, Startpage deliberately resells Google, and Dogpile buys feeds. Hybrids exist too — DuckDuckGo runs its own crawler but takes web links from Bing. Reselling is not a criticism; it is a product decision, and for Startpage it is the product.
P
PageRank
The link-analysis algorithm published by Larry Page and Sergey Brin in 1998, which scores a page's importance from the quantity and quality of links pointing to it, recursively: a link from a page that is itself well linked counts for more. It is usually described as a random surfer's probability of landing on a page. PageRank matters because it was the decisive competitive advantage of early Google over engines that ranked mainly on term frequency — AltaVista had a larger index and better query syntax but no link-graph ranking, and lost on relevance. It has not been a standalone ranking factor for many years, and the public toolbar score was retired in 2016, but the idea of links as votes remains foundational.
Paid results also: Sponsored listings, Search ads, Paid search
Listings that appear on a results page because an advertiser bid on the query. The mechanism was invented by GoTo.com, launched by Bill Gross in 1998, which auctioned position on a keyword and charged per click; renamed Overture, it syndicated those listings into Yahoo, MSN, AOL and Excite and was bought by Yahoo in 2003. Google adapted the model by weighting bids with predicted click-through rate, and search advertising became the revenue engine of the entire consumer web. It matters here because it explains almost every structural feature of search: why engines are free, why they want your query data, why results pages have grown denser, and why subscription engines such as Kagi exist as a deliberate counter-model.
Personalised results also: Personalisation, Tailored results
Adjusting what an engine returns according to what it knows about the individual asking — previous searches, clicks, account history, location, language and device. Personalisation is a spectrum, and the ends are very different: location and language adjustment is nearly universal and largely uncontroversial, while ranking from a behavioural profile requires the engine to keep a history linked to a person. It matters because it makes results non-reproducible, which is a problem for journalists, researchers and anyone comparing engines — two people cannot check whether they saw the same thing. Several engines refuse it outright as a design principle: Mojeek returns identical results to every user for the same query, and states the absence of a filter bubble as a feature.
Politeness delay also: Crawl-delay, Crawl rate limiting
The interval a crawler waits between successive requests to the same host, so that indexing the web does not amount to overwhelming the servers that host it. Politeness is a per-host constraint rather than a global one — a crawler may fetch thousands of pages a second overall while touching any single site slowly — and it is a large part of why crawling the web takes so long. Some engines publish their limits: Mojeek documents no more than one request per second to a given site. The non-standard Crawl-delay directive in robots.txt is honoured by some crawlers and ignored by others, including Googlebot, which schedules its own rate instead.
Precision and recall also: Relevance metrics
The two standard measures of retrieval quality. Precision is the share of returned results that are relevant; recall is the share of all relevant documents that were returned. They trade against each other — a system can guarantee perfect recall by returning everything, at ruinous precision — and web search resolves the trade-off decisively in favour of precision on the first few results, because nobody reads page four. That choice explains much of what search feels like: engines rewrite, expand and filter queries to keep the top of the page clean, at the cost of completeness. Recall-oriented retrieval survives where completeness is the requirement, in patent, legal and systematic-review searching, which is one reason those fields still use Boolean syntax.
Q
Query also: Search query, Search term
The words a person enters, or speaks, to ask an engine for something. A query is not the same as a keyword: one underlying need is expressed in many phrasings, and engines see enormous variety — Google has said for years that a substantial share of daily queries have never been seen before, which is why matching against a fixed list of terms was never sufficient. Queries are also the engine's most valuable data: they are what advertising is sold against, what autocomplete is built from, and what click-through data attaches to. That is why privacy-oriented engines describe their handling of them so precisely, and why SearXNG originally submitted queries by HTTP POST to keep them out of intermediary server logs.
Query understanding also: Query interpretation, Query rewriting
Everything an engine does to a query between receiving it and searching the index: correcting spelling, segmenting words, detecting language and location, classifying intent, expanding terms to synonyms, dropping or reweighting words, and deciding which result formats the question deserves. Almost none of it is visible, and it is the reason two engines holding similar documents return visibly different pages. Query understanding is also why exact-match searching has become something you must specifically request: by default a modern engine answers the question it believes you asked, not the string you typed. Google's Hummingbird update in 2013 was an early public marker of the shift, and BERT and later language models are its current machinery.
R
RankBrain
A machine-learning system Google introduced in 2015 to help interpret queries it had not seen before, by representing words and phrases as vectors and finding queries with similar meanings whose results were already understood. Google described it at the time as one of its most important ranking signals. RankBrain matters as a marker rather than as a mechanism: it was the first widely acknowledged use of machine learning inside Google's core ranking, and it established the pattern of ranking systems that are trained rather than written. Its successors — BERT from 2019, and later neural ranking and generative systems — followed the same path, and none of them is documented in enough detail to be reproduced from outside.
Ranking signal also: Ranking factor
Any piece of evidence a ranking system uses to order results — term matches in a document, link-graph scores, language and location match, freshness, page-level quality classifications, and on engines that collect it, aggregated user interaction. Two things about signals are routinely misstated. First, published lists of them are inference: engines confirm broad families of signals and almost never confirm weights, and the confirmed set changes. Second, signals are not independent knobs; modern systems learn combinations, so a signal's effect depends on the query. This site describes signals to explain how ranking works, not to advise anyone on how to move within it. Where an engine has documented its own systems by name — Naver's C-Rank, for instance — that documentation is a vendor claim.
Redirect also: 301 redirect, 302 redirect, HTTP redirect
An HTTP response that tells a client the resource is at a different URL. The two that matter for search are 301 Moved Permanently, which states the new address is the address from now on, and 302 Found, which states the move is temporary and the original should be kept. Engines treat the pair very differently at indexing time: a 301 consolidates the old URL into the new one, while a 302 is a reason to hold on to the original. Redirects matter because the web is not stable — sites move, merge and change domains constantly — and a chain of redirects is often the only surviving evidence connecting an old address to a live document. Redirect loops and long chains are a common cause of pages disappearing from indexes.
Resold results also: White-label results, Rebranded results, Third-party feed
Results an engine did not produce, licensed from an engine that did and presented in its own interface. It is one of the three structural positions a search product can occupy — own index, hybrid, or resold — and it is the most common by far among the alternatives people are offered. Startpage resells Google and says so openly, its product being the privacy layer rather than the ranking; Yahoo! has been served by Bing since 2010; Kiddle is a Google Programmable Search instance with SafeSearch and its own blocklists. Reselling is not a criticism, but it bounds what an engine can be: a reseller cannot find a page its supplier has not indexed, and its independence lasts as long as the contract does.
Rich result also: Rich snippet, Enhanced result
A search result rendered with additional detail drawn from a page's structured data — star ratings, cooking times, event dates, product availability, breadcrumb trails. The engine parses schema.org markup, usually as JSON-LD, and if the page qualifies under its own rules it displays the extra fields. Rich results matter because they change what a results page is made of: the listing carries facts rather than only a title and an extract, and the reader may not need the page. They are also entirely at the engine's discretion. Markup is an input, never a guarantee, engines withdraw support for result types when they stop being useful, and unsupported or contradicted claims in markup are ignored.
Robots.txt also: Robots Exclusion Protocol, REP
A plain-text file at the root of a host — /robots.txt — that tells automated clients which paths they may fetch. It was proposed by Martijn Koster in 1994 as an informal convention and only standardised as RFC 9309 in 2022. Rules are grouped by User-agent and expressed as Disallow and Allow paths. Two limits define what it is: it governs crawling, not indexing, so a disallowed URL can still be listed on the strength of links to it; and compliance is voluntary, which is why the file has become the front line of the argument over AI training crawlers. Mojeek documents how MojeekBot matches records; Google publishes tokens for each of its crawlers.
S
SafeSearch also: Safe search filtering, Content filtering
A setting that suppresses explicit results, most familiar as Google's SafeSearch but implemented in some form by most major engines. It is a filter applied over an existing index rather than a separate index, which is what makes it available to third parties: Kiddle, for instance, is a Google Programmable Search instance with SafeSearch enabled plus its own domain and keyword blocklists and hand-picked top results. Filtering matters because it is where an engine makes explicit editorial decisions and can be examined on them — and where those decisions go wrong visibly. Kiddle was reported in March 2016 to be blocking searches for terms including gay, lesbian and transgender, a filtering choice, not a technical accident.
Schema markup also: Structured data, schema.org
A shared vocabulary for describing what a page is about in machine-readable form — that this is a Recipe, that this string is its author, that this number is a ratingValue. schema.org was launched in June 2011 by Google, Microsoft and Yahoo, later joined by Yandex, precisely so that publishers would not have to mark up separately for each engine, and it is maintained as an open vocabulary. It matters because HTML describes presentation, not meaning: without a vocabulary an engine must infer from prose that a page describes an event on a date. Markup is usually written as JSON-LD, and engines treat it as evidence to be checked against the visible page rather than as fact.
Search API also: Search engine API, Results API
Metered programmatic access to an engine's index, sold to developers rather than shown to users. APIs have become strategically important because AI products need grounding data and very few organisations own an index to ground on. The market shifted abruptly on 11 August 2025, when Microsoft retired the public Bing Search APIs entirely, directing developers to the more expensive Grounding with Bing Search within Azure — every product built as a Bing reseller had to move. That scarcity is why independent indexes now sell access as a core business: Brave launched its Search API in May 2023, Mojeek licenses its index, and Perplexity opened a Search API in September 2025, which is only possible because it owns an index.
Search engine also: Web search engine
A system that finds documents matching a query from a stored index of the web. Three components define it: a crawler that fetches pages, an index that stores and organises them, and a ranking system that orders what matches. Anything lacking all three is doing something else — a directory such as DMOZ catalogued sites by hand, a meta-search engine such as Dogpile borrows other engines' results, and Wolfram Alpha computes answers from curated data without crawling at all. The definition matters because the word is applied loosely to any box that accepts a query, which obscures the question a reader usually wants answered: whether the thing has an index of its own or is showing someone else's.
Search operators also: Advanced operators, Search syntax
Special tokens that constrain a query beyond its words — site: to limit results to one host, filetype: to one document format, intitle: to a field, - to exclude a term, quotation marks to require a phrase, and date or language restrictions. They matter because they turn a relevance guess into something closer to a database query, which is what researchers, journalists and archivists need. Support has narrowed over time: AltaVista and Gigablast offered full field and Boolean syntax, while mainstream engines now treat several operators as advisory and have quietly dropped others, including Google's cache: operator in 2024. Documented syntax differs by engine, so an operator that works on one may be ignored by another.
Search syndication also: Results licensing, Syndication deal, Feed deal
The wholesale business of search: one engine licenses its results, its advertisements, or both to another, which shows them under its own brand and shares the revenue. Syndication is why the number of visible search brands is far larger than the number of indexes. Microsoft supplies web results to Yahoo!, DuckDuckGo and Ecosia among others; Dogpile and Excite buy feeds through System1; GoTo.com built the entire paid-search industry by syndicating listings into Yahoo, MSN, AOL and Excite. It also became a competition remedy: the US court's September 2025 decision required Google to license search results and text ads to qualified competitors on five-year terms, initially capped at 40% of queries.
searchAction also: SearchAction, WebSite SearchAction, Sitelinks searchbox markup
A schema.org type describing the act of searching, most often used as a potentialAction on a WebSite object to declare that a site has its own search and how to call it. The pattern gives a target URL template with a placeholder — for example https://example.com/search?q={search_term_string} — and a query-input property marking that placeholder as required, so a consuming program knows how to construct a search request for that site. Google used this markup to power its sitelinks search box, a search field shown inside a brand's result. Google announced in October 2024 that the feature would be retired, removed it on 21 November 2024, and stated the markup was no longer used for it. The schema.org type itself remains valid and is not Google-specific.
Semantic search also: Meaning-based search
Retrieval that matches on meaning rather than on the presence of the query's exact words, so a page can be returned without containing any of them. It is implemented with embeddings, entity recognition and language models that map queries and documents into a shared representation of meaning. Semantic search matters because it fixes the oldest failure of keyword systems — the same idea expressed in different vocabulary — and it is what makes conversational questions workable. It introduces the opposite failure in exchange: results that are about the right subject but not about the thing asked, and no way for the user to see why a document was retrieved. That trade-off is exactly why literal search remains a necessary escape hatch.
SERP also: Search engine results page
The page an engine returns for a query. The term dates from a time when the page was a list of ten links with a title, URL and extract; it now describes a composite in which ranked web documents are one element among ads, an AI summary, a knowledge panel, a local pack, images, video, shopping units and People Also Ask questions. That change is the substantive fact about modern search: the proportion of the page consisting of links to other people's websites has fallen, and the proportion consisting of the engine's own content has risen. SERP composition varies by engine as much as ranking does — Naver stacks its own blog, community and Q&A blocks above general web results.
Sitemap also: XML sitemap, sitemaps.org
A file listing a site's URLs for crawlers, usually XML following the sitemaps.org protocol, which Google, Yahoo and Microsoft adopted jointly in 2006. Each entry may carry a last-modified date and a change frequency. A sitemap supplements link discovery rather than replacing it: it tells an engine that a URL exists, which helps most for large sites, newly launched sites, and pages that few links point to. It does not require an engine to crawl or keep anything, and inclusion in a sitemap is not a claim an engine has to accept. Sitemaps are also how many engines learn about pages before any external link to them exists, which is why submission interfaces such as Naver's Search Advisor accept them.
Snippet also: Result description, Extract
The short passage of text shown beneath a result's title, intended to show why the page matched. Snippets are generated by the engine from the stored document at query time, usually by selecting the passages containing the query terms, which is why the same page shows different text for different queries. They may be drawn from a page's meta description, from its body text, or from structured data, at the engine's discretion. Snippets matter more than their size suggests: they are the entire basis on which a user decides which of ten results to open, and long snippets are the mechanism behind featured snippets and much of the zero-click argument between engines and publishers.
Stemming also: Suffix stripping, Lemmatisation
Reducing words to a common root so that variants match — running, runs and ran treated as forms of run. The classic implementation is the Porter stemmer, published by Martin Porter in 1980, which strips suffixes by rule; lemmatisation is the more careful relative that uses a dictionary and part-of-speech analysis to find the true base form. Stemming matters because it is the difference between a query returning what a user meant and returning only what they typed, and because it is applied at indexing time, which means it is baked into what the index can retrieve. It also causes characteristic errors — over-stemming collapses unrelated words — and it is one of the transformations a literal search switches off.
Stop words also: Function words, Noise words
Extremely common words — the, of, and, is — that early retrieval systems removed from queries and indexes because they appear nearly everywhere and so carry little information about which document matches. Dropping them saved substantial storage when that mattered. Modern engines mostly do not discard them, because word order and function words often carry the meaning of a question: Google gave the removal of a small word changing the sense of a query as one of the motivating examples for deploying BERT. The concept persists in weighting rather than deletion — TF-IDF and BM25 both give near-zero weight to terms appearing in almost every document, achieving the same effect without throwing information away.
T
TF-IDF also: Term frequency-inverse document frequency
A weighting scheme that scores a term's importance to a document as its frequency in that document multiplied by how rare it is across the collection. The second half is the insight: a word appearing in nearly every document distinguishes nothing, while a rare word appearing several times in one document is strong evidence about what that document is about. Inverse document frequency was introduced by Karen Sparck Jones in 1972 and the vector-space model built around it by Gerard Salton's group. TF-IDF matters because it is the ancestor of every lexical ranking function since, BM25 included, and because it explains a persistent property of search: rare and specific queries are easier for an engine to answer well than common ones.
Tokenisation also: Word segmentation, Text analysis
Splitting text into the units that will be stored in the index. In English it looks trivial until the cases arrive — hyphenation, apostrophes, URLs, product codes such as C++, and numbers — and each decision determines what can later be found. In Chinese, Japanese and Thai it is a hard problem in itself, since those scripts do not separate words with spaces, so an engine must infer word boundaries before it can index anything; this is one reason domestic engines such as Baidu, Naver and Sogou hold an advantage in their own languages. Tokenisation matters because it happens once, at indexing time, and no clever query can recover a distinction the tokeniser discarded.
U
User-agent also: UA string, Crawler identity
The identifying string a client sends with every HTTP request, and the token by which robots.txt rules address a particular crawler. Published, stable user-agents are what make a crawler accountable: they let a site operator see who is fetching, apply rules to that crawler specifically, and verify the identity by reverse DNS or against a published IP list, as Mojeek and Google both document. Their absence is a real dispute. Brave's crawler help pages have at times stated that it did not advertise a differentiated user-agent, on the grounds that sites allowing only Google to crawl would otherwise exclude it — leaving publishers unable to block it cleanly. The declared Bravebot/1.0 token resolves that.
V
Vertical search also: Specialised search, Domain-specific search
Search restricted to one kind of content or one subject area, rather than the whole web: flights, jobs, code, academic papers, patents, products, films. Vertical engines can do things a general engine cannot, because their data is structured — they can sort by price, filter by publication date, or search fielded records — and because their ranking can be tuned to a single kind of need. The category matters commercially because the most valuable queries are vertical, which is why general engines have absorbed so many of them into their own results pages, and it matters structurally because much vertical content sits in the deep web where crawlers cannot reach it. Wolfram Alpha is the extreme case: curated data and computation, no crawl.
Voice search also: Spoken search, Voice assistant search
Queries spoken to a device and answered aloud, through assistants such as Google Assistant, Siri and Alexa. Two things distinguish it. Spoken queries are longer and more conversational than typed ones, and they arrive as transcribed text, so speech recognition errors become retrieval errors — proper nouns and homophones are the usual casualties. More importantly, the answer is a single spoken response rather than a page of choices, so voice collapses the results page to one result, typically the source an engine would have shown as a featured snippet. That makes voice the earliest large-scale example of the pattern AI answers have since generalised: an answer given without the user seeing the alternatives.
W
Walled garden also: Closed ecosystem
A search product that answers queries mainly with content its operator owns or hosts, rather than sending users out to the open web. Naver is the clearest case and a deliberate one: facing very little Korean-language content to index in 1999, it built the content instead — Knowledge iN, Blog, Cafe — and stacks those blocks above general web results, a self-preferencing that the Korea Fair Trade Commission fined it for in shopping and video search in 2020. Baidu faced the same criticism in a widely read 2019 essay arguing it mostly returned its own Baijiahao content. Walled gardens matter for measurement as well as principle: searches answered without leaving the site produce no referral, so tracker-based market-share estimates undercount them.
Z
Zero-click search also: No-click search
A search that ends on the results page, because the answer was there — in a featured snippet, a knowledge panel, a weather or currency box, or an AI Overview — and no result was opened. The pattern predates generative AI and has grown with each new answer format. It matters because it goes to the bargain that funded the open web: publishers accepted crawling because search sent readers back, and an answer drawn from a page without the visit removes that return. Published estimates of how large the share is vary widely and are contested, because they depend on measurement method and query mix; the direction is better evidenced than any particular figure, and figures should be quoted with their source and date.