The history of search engines

Search is older than the web

The first internet search engine predates the public World Wide Web. Archie went live on 10 September 1990, written at McGill University in Montreal by Alan Emtage with Bill Heelan and Peter J. Deutsch. It catalogued the filenames held on public anonymous-FTP servers: you gave it a name or part of one and it told you which hosts had a file called that, and where. It read nothing inside those files, ranked nothing and summarised nothing — you had to know roughly what the thing you wanted was called.

Sources disagree on the year: McGill and the Internet Hall of Fame date Archie's creation to 1989, while Wikipedia's timeline gives 10 September 1990 for the service going live. Emtage was inducted into the Internet Hall of Fame in 2017, credited with "the world's first Internet search engine".

Gopher, the menu protocol that briefly rivalled the web, produced Veronica and Jughead, which searched menu titles the way Archie searched filenames. All three shared the limitation that defined the era: they searched labels. Nobody yet searched the text of documents.

1993 to 1995: the crawler arrives, and everything changes

The first attempt to index the web did not crawl it. ALIWEB — "Archie-Like Indexing in the WEB" — was announced by Martijn Koster in late 1993 and presented at the first International World Wide Web Conference at CERN in May 1994. Site owners wrote a small structured file describing their own pages and registered it; ALIWEB fetched those files and merged them. It failed for the reason every submission-based system fails: almost nobody submitted. Koster went on, that same year, to write the Robots Exclusion Standard — the robots.txt convention still in use today.

The break came on 21 April 1994, when Brian Pinkerton launched WebCrawler from the University of Washington with a database of just over 4,000 sites. It was the first engine to index the full text of the pages it fetched rather than their titles or descriptions, which meant that for the first time you could search for a phrase you remembered from inside a document. It answered its millionth query on 14 November 1994.

The rest arrived in eighteen months. Lycos, Michael Mauldin's research project at Carnegie Mellon, went online on 20 July 1994 with 54,000 documents and had over 634,000 by the end of August. Infoseek launched in January 1994 as a pay-per-use service, dropped the paid model that August and relaunched free and advertising-supported in February 1995 — an early demonstration of which business model search would take. Excite launched in October 1995. And on 15 December 1995 Digital Equipment Corporation's Palo Alto research labs launched AltaVista, for roughly three years the fastest and most comprehensive full-text search on the web.

The directory era, and why humans lost

Running alongside the crawlers was an entirely different idea: catalogue the web by hand. Yahoo began in 1994 as "Jerry and David's Guide to the World Wide Web", a hierarchy of hand-picked sites kept by Jerry Yang and David Filo at Stanford. LookSmart, founded in Melbourne in 1995 and relaunched under that name in October 1996, did the same commercially.

The most ambitious version was DMOZ, launched on 5 June 1998 as GnuHoo, renamed NewHoo after objections, acquired by Netscape that October and hosted at directory.mozilla.org — hence the name. Volunteers applied to edit a category and decided which sites deserved a listing. Its data was freely licensed, which is why it ended up underneath a great many other products, Google Directory among them.

Directories lost for two reasons: the web grew faster than volunteers could read it, and human editing introduced the one thing a catalogue cannot survive, a queue. Backlogs of months or years in commercial categories were routine, and allegations that editors favoured their own sites were persistent. Google shut Google Directory in July 2011, removing DMOZ's largest distribution channel; DMOZ itself closed on 17 March 2017, after a notice in early March had given the date as 14 March. Both dates circulate.

Yahoo, meanwhile, had solved its own problem by buying results: from AltaVista in 1996, then Inktomi, then Google from 2000 to 2004. For four years the search box on the biggest portal on the internet was Google's — the origin of most of the perennial confusion about who powers whom.

1997 to 2000: ranking becomes the product

By 1997 finding pages was no longer the hard part; sorting them was. AltaVista could return a hundred thousand matches with no principled way to decide which ten belonged at the top.

The answer came out of a Stanford project called BackRub, run by Larry Page and Sergey Brin in 1996 and 1997, which treated a hyperlink as a vote and weighted each vote by the linking page's own score. The google.com domain was registered on 15 September 1997 and Google Inc. was incorporated on 4 September 1998. PageRank was not the only good idea of the period — Teoma, out of Rutgers research and launched in April 2001, ranked instead by the authority of pages within the topic of the query — but it was the one that shipped at scale.

The same window produced the engines that still dominate outside the English-speaking web, and American histories routinely omit them. Yandex launched at yandex.ru on 23 September 1997, founded by Arkady Volozh and Ilya Segalovich on morphological work with Russian dating back to 1990. Seznam started in Prague in 1996 as a directory of Czech sites, adding fulltext search on its own technology in 2005. Naver launched on 2 June 1999 as the first Korean service with its own search engine, and in August 2000 introduced 통합검색, "comprehensive search" — a results page divided into typed blocks by content category, still the shape of a Naver page today. Baidu was founded in Beijing on 18 January 2000 by Robin Li, who had already patented a hyperlink-analysis ranking method called RankDex, and Eric Xu.

How search learned to pay for itself

None of this had a business model until Bill Gross supplied one. In February 1998 his Idealab company GoTo.com launched an open auction in which advertisers bid for placement against a keyword and paid only when someone clicked. He presented it at TED8 on 21 February 1998 to an audience described as confused and in places hostile at the idea of results being openly sold; unpaid listings were backfilled from Inktomi from June 1998.

GoTo did not win as a destination. It won as plumbing, syndicating its paid listings into Yahoo, MSN, AOL and Excite for a share of the revenue. It filed the defining patent on 28 May 1999 — granted as US 6,269,361 on 31 July 2001 — renamed itself Overture Services in October 2001, and sued Google in April 2002 over AdWords Select, the February 2002 product that replaced Google's per-impression pricing with a keyword auction. Yahoo bought Overture in 2003 for $1.63 billion; the litigation settled with Google taking a licence and handing Yahoo shares immediately before its 2004 flotation.

Every search advertisement since descends from that 1998 experiment, and it frames everything that follows: from 1998 onward, a general search engine was an advertising business that happened to run a crawler.

The extinction event, 1999 to 2013

The first generation did not fade. It was bought at absurd valuations and then written off.

• Excite merged with @Home Network in January 1999 in a deal valued at $6.7 billion in stock, filed for Chapter 11 on 1 October 2001, and had its portal sold to iWon and InfoSpace that December for roughly $10 million.
• Lycos was acquired by Terra Networks in October 2000 for $12.5 billion, sold to Daum in August 2004 for $95.4 million and sold on in 2010 for $36 million. It retired its own crawler in autumn 2001.
• Infoseek was absorbed into Disney's GO.com. Its crawler ran for the last time in January 2001, and Disney announced the shutdown on 29 January 2001 — around 400 jobs and a write-off it put at roughly $790 million in its SEC filing.
• Magellan was closed by Excite in May 2001 and Northern Light withdrew its free public web engine in January 2002.
• AltaVista was sold to Overture in February 2003 for about $140 million, passed to Yahoo months later and shut down on 8 July 2013. AllTheWeb, acquired in the same 2003 sweep, was closed by Yahoo on 4 April 2011.
• Ask Jeeves bought Teoma in September 2001 for a little over $1.5 million — a working crawler and a novel ranking algorithm, and one of the great bargains in search. IAC paid $1.85 billion for Ask Jeeves in 2005, dropped the butler in February 2006 and shut the index in late 2010. Ask.com itself closed on 1 May 2026.

Yahoo's own trajectory is the clearest illustration. It launched its own crawler and index in February 2004, after a decade of buying results, and ran them for about six years. On 29 July 2009 it signed the Search Alliance with Microsoft, retired its index, and has served Bing's web results ever since.

Note what survived. Excite, Lycos, HotBot, WebCrawler, MetaCrawler and Dogpile are all still reachable in 2026, and none is a search engine: they are legacy domains under IAC or System1 with a search box wired into somebody else's feed — brands outliving the technology that made them.

2009 to 2022: two indexes, many interfaces

Bing launched on 3 June 2009 and, with the Yahoo alliance the following month, settled the shape of Western search for fifteen years: two organisations crawling the web at scale, everyone else licensing from one of them.

That is the period in which the alternatives appeared. DuckDuckGo launched in 2008 and grew sharply after the 2013 Snowden disclosures, though its web links came from Bing then and come from Bing now. Startpage was reconfigured on 7 July 2009 to relay queries to Google without identifying the user — a privacy proxy, deliberately, with no index of its own.

Genuinely independent indexes were rarer and smaller. Mojeek, started in 2004 and public from 2006, has never resold anyone's results. Gigablast ran its own crawler for some twenty-one years before going offline without announcement in early April 2023. Cuil launched in July 2008 to enormous publicity and collapsed by 17 September 2010. Brave Search entered public beta on 22 June 2021 and Marginalia's first commit was 26 February 2021; Neeva, an ad-free subscription engine with its own crawl, shut its consumer product on 2 June 2023.

2023 to 2026: answers on top of links

On 7 February 2023 Microsoft put a GPT-4-based chatbot on the Bing results page — the first mainstream integration of a large language model into a search engine, about three months ahead of Google's Search Generative Experience in May 2023. AI Overviews reached all US users in May 2024 and more than 100 countries on 28 October 2024; AI Mode followed in March 2025; Gemini 3 became the default model behind AI Overviews globally on 27 January 2026. OpenAI launched ChatGPT search on 31 October 2024, and Perplexity had shipped cited AI answers as its entire product on 7 December 2022.

Two structural events sit underneath the product news. Microsoft retired every public Bing Search API on 11 August 2025, cutting off the cheap route by which small engines and tools had resold Bing results and replacing it with a more expensive AI-grounding product. And Judge Amit Mehta ruled on 5 August 2024 that Google had unlawfully maintained a monopoly in general search; the remedies decision of 2 September 2025 declined to order a Chrome divestiture but banned exclusive default deals and required Google to share search-index and click-and-query data with qualified competitors and to license its results. Those remedies took effect on 3 February 2026 and are under appeal by both sides at the D.C. Circuit, undecided as of August 2026.

The map kept changing at the edges. Ecosia and Qwant's joint venture began serving live traffic from its own European index, Staan, in August 2025. Yahoo launched an AI answer engine, Scout, on 27 January 2026 — grounded on Bing. Ask.com, thirty years old, closed on 1 May 2026: "Every great search must come to an end."

The arc is legible. Search began as a way to find filenames, became a way to find documents, then a way to rank them, then an advertising auction attached to that ranking, and is now a way to be told the answer. What has not changed since 1994 is the requirement underneath all of it: somebody has to crawl the web and keep a copy of it, and remarkably few organisations ever have.

Frequently asked questions

Sources

• en.wikipedia.org/wiki/Timeline_of_web_search_engines
• en.wikipedia.org/wiki/WebCrawler
• en.wikipedia.org/wiki/AltaVista
• en.wikipedia.org/wiki/GoTo.com
• patents.google.com/patent/US6269361B1/en
• en.wikipedia.org/wiki/DMOZ
• everycrsreport.com/reports/LSB11362.html
• learn.microsoft.com/en-us/lifecycle/announcements/bing-search-api-retirement