A network with no map
By the late 1980s, the precursor networks to the modern internet already held a meaningful amount of content — software packages, research papers, documentation — scattered across thousands of FTP (File Transfer Protocol) servers with no central catalog. Finding a specific file meant already knowing which server held it, or asking on a mailing list and waiting for someone to reply. There was no equivalent of typing a query into a box.
Archie: the first search engine
In 1990, Alan Emtage, then a student at McGill University in Montreal, built Archie (a shortening of "archive") to solve exactly that problem. Archie periodically downloaded directory listings from public FTP servers and compiled them into a single searchable database of filenames. It did not look inside files or understand their content — it only matched filenames against a user's query — but that was a genuine breakthrough. For the first time, someone could search a meaningful slice of the network from one place rather than crawling it by hand.
Archie is widely credited as the first true search engine, predating the World Wide Web itself by roughly a year. It also established a pattern that has held ever since: a search engine needs an automated process to gather data (today we call this crawling), and a structured store of what it found (today, an index).
Veronica and Jughead
As Gopher — a menu-driven document retrieval system that predates the web's hyperlink model — grew in popularity in the early 1990s, two tools extended Archie's filename-indexing approach to Gopher space: Veronica (Very Easy Rodent-Oriented Netwide Index to Computerized Archives) and Jughead (Jonzy's Universal Gopher Hierarchy Excavation And Display). Both, in the tradition of the era's search tools, took their names as a joke on Archie Comics. Functionally, they did for Gopher menus what Archie did for FTP filenames: index titles so users could search rather than browse.
The first web crawlers
The web itself, publicly introduced in 1991, needed its own discovery tools almost immediately. The World Wide Web Wanderer, built by Matthew Gray at MIT in 1993, is generally considered the first robot deployed on the web, initially built to measure its size rather than to power a search tool — but the crawling technique it pioneered became the foundation for every search engine that followed.
WebCrawler, launched in 1994 by Brian Pinkerton, was the first search engine to index the full text of web pages rather than just their titles or URLs. This was a meaningful leap beyond Archie's filename matching: for the first time, a search engine could tell you a page was relevant because of what it actually said, not just what it was called. WebCrawler's approach — full-text indexing of crawled pages — is the direct ancestor of how every modern search engine still works at a basic level, however much has been layered on top of it since.