Guides

Guides to search engines and how they work

Four sections: the mechanics of crawling, indexing and ranking; the history of search from 1990 onward; how to choose an engine and tell an independent index from a resold one; and what the shift to AI answers actually changed.

Start here

How search engines work

Crawling, indexing, ranking and the results page, explained with real crawlers and real index sizes.

The history of search engines

Archie, WebCrawler, AltaVista, Google, Bing and the AI answer engines — the whole arc, with dates.

Choosing a search engine

Own index, hybrid, or resold — why most Google-alternative lists are misleading, and how to check for yourself.

AI search and the rise of answer engines

Retrieval versus generation, who owns the index behind the answer, and what a citation actually proves.

How search engines work

12 guides

What actually happens between typing a query and seeing a page of results — crawling, indexing, ranking, and who pays for it.

What is a search engine?

The three-part definition, and the test that separates a real search engine from a search box over someone else's index.

How search engine crawling works

The fetch loop, the URL frontier, crawl rate limits, recrawl cadence and how engines prove a crawler is really theirs.

How search engine indexing works

Inverted indexes, tokenising, selection and sharding — what happens between a crawler fetching a page and a query matching it.

How search ranking works

Term frequency, link analysis, learned rerankers and personalisation: how engines decide which twenty results a person sees.

What is a web crawler?

The named crawlers you actually see in server logs, what each one is for, and how engines let you verify them.

Robots.txt and crawler control

Where robots.txt came from, what it specifies, why it governs crawling not indexing, and how AI crawlers treat it.

How search engines handle duplicate pages

Why the same page exists at a dozen addresses, and how engines cluster near-duplicates and pick one to keep.

How big are search engine indexes?

The published numbers, what they actually count, and why no outsider can verify the size of a search index.

What is a SERP?

The anatomy of a search engine results page: organic listings, ads, features, and why yours differs from everyone else's.

SERP features explained

Answer boxes, question accordions, entity panels and AI summaries: what each is, and where the text comes from.

How search engines make money

Advertising, subscriptions, API metering, affiliate commissions and mission funding — and what each does to the results.

How search autocomplete works

Prefix matching, query logs and millisecond budgets: how engines guess the rest of your query, and why yours differs.

The history of search

9 guides

From FTP file listings in 1990 to AI answers — how search got here, and which engines died on the way.

What was the first search engine?

Archie in 1990, WebCrawler in 1994. Six tools are called the first search engine, and each claim is to a different title.

Searching the internet before the web

Archie, Veronica and Jughead searched FTP filenames and Gopher menus years before any tool indexed a web page.

The directory era of web search

Yahoo, Magellan, LookSmart and DMOZ catalogued the web by hand. Why human curation lost, and what survived it.

Google and PageRank

The 1998 paper, the random-surfer formula, the green toolbar bar, and what survives of PageRank inside Google today.

Search algorithm updates: a history

What each named algorithm update actually changed, why the names stopped, and who inherits updates they did not make.

Mobile, local and voice search

Mobile-first indexing, the business directory behind local results, and the road from GOOG-411 to the assistant on your phone.

Search engines that no longer exist

Dead engines with dates and causes, separated from the brands that still resolve but stopped being search engines years ago.

A timeline of search engines, 1990 to 2026

Thirty-six years of search, dated: launches, acquisitions and shutdowns from Archie in 1990 to Ask.com in 2026.

The search engine wars

Yahoo, Excite, Lycos, AltaVista, GoTo.com and Google — who fought over what, and which fight actually decided it.

Choosing a search engine

14 guides

Which engines run their own index, which resell someone else's, and how to tell the difference before you switch.

Own index vs resold results: who actually crawls the web

The handful of search engines that crawl the web themselves, the many that resell Bing or Google, and how to tell them apart.

Search engines other than Google

The engines that are genuinely different from Google, the ones that only look different, and how to choose between them.

Private search engines compared

The four questions that separate private search engines, applied to every major one — and the claims that are unverified.

Search engines that don't use AI

The engines with no AI answer layer, the ones with a real off switch, and the Google workaround — checked August 2026.

Unbiased search engines: what is actually possible

Why no search engine is unbiased, which kinds of bias are checkable, and which engines let you see or change the ranking.

How many search engines are there?

Why there is no true count of search engines, and why independent web indexes are far fewer than the brands.

Meta search engines explained

How metasearch works, the founding generation of the 1990s, the InfoSpace lineage, and the modern open-source revival.

Advanced search operators explained

How search operators work, which are portable across engines, which have quietly stopped working, and what they cannot do.

Kids and safe search engines: what they really are

Why kids search engines are wrappers over Google or Bing, how their filters work, and what the documented record shows.

People search engines explained

How people-search sites build profiles from public records, marketing data and scraped pages, and why so much of it is wrong.

Search engine market share, measured and dated

Worldwide search share for July 2026 per StatCounter, the device split, and why the headline figure is a proxy not a census.

Search engine market share by country

Country-by-country search share for July 2026, the markets Google does not lead, and where the measurement itself breaks down.

Search engine comparison: index, ranking, privacy and money

Twelve engines compared on index source, ranking, privacy and monetisation — grouped by who actually owns the index.

Search engine APIs and developer tools

Who still sells programmatic access to a web index after the Bing retirement, and how the billing is structured.

The shift to AI search

6 guides

Answer engines, AI Overviews and generated summaries — what changed, what did not, and what it means for the open web.

What is an answer engine?

Answer engines return a written answer with citations rather than ranked links — what they are and how they differ from search.

AI Overviews explained

Google's AI Overviews: what they are, how they are generated from Google's own index, and the arguments they started.

Conversational search engines

Search products you hold a conversation with: how they work, which ones exist, and whose web index each is grounded on.

AEO and GEO explained

What AEO and GEO claim to be, where the terms came from, and what the platforms themselves say about them.

How AI search engines cite sources

Where citations in AI answers come from, what they prove, and why the citable set is fixed by whose index the engine uses.

AI search vs traditional search

What changed when search started generating answers, what stayed the same underneath, and how the two compare structurally.