Searching the internet before the web

The internet had search engines before it had a web

The first internet search engine, Archie, went live on 10 September 1990 — roughly a year before the World Wide Web became publicly available. For the next three or four years the internet was searchable and the web was not, and the tools that did the searching worked on a completely different principle from anything that came after.

This is the single fact that makes the whole first search engine argument tractable. The internet is the network: it carried FTP, email, Usenet, Telnet and Gopher for years before it carried HTTP. The web is one service running on top of it, and it arrived late. So a search engine built in 1990 was not a worse version of Google; it was a tool for a different kind of content, and it was shaped end to end by what that content looked like.

What it looked like was files on servers. Not documents linked to other documents, but archives — software, papers, images, mailing-list digests — sitting in directories on machines that let anonymous users connect and download. There were no hyperlinks between them, so there was nothing to crawl. There was no page text to index, because most of the content was not text. What every server did have was a list of what it was holding, and that turned out to be enough to build a search engine on.

Archie — searching FTP filenames

Archie was built at McGill University in Montreal by Alan Emtage, with Bill Heelan and Peter J. Deutsch. It solved a real and specific problem: by 1990 there were hundreds of public anonymous-FTP servers, and finding which one held a given file meant connecting to each in turn and looking around.

Its mechanism was a scheduled harvest rather than a crawl:

1. Archie maintained a list of known public anonymous-FTP sites — a list that was itself maintained by hand, because there was no way to discover new hosts automatically.
2. Roughly monthly, it connected to each site and pulled down full recursive directory listings: hostname, path, filename, size, date.
3. Those listings were merged into one central searchable database of strings.
4. Users matched against that database by exact filename, by substring, or by regular expression, depending on the client and server.

Note what is missing. No link graph, because there were no links. No relevance ranking, because there was no notion of relevance — a filename either matched or it did not. And crucially, no idea what any file contained. Archie could tell you that a file called tcpdump-3.4.tar.Z existed on eleven hosts. It could not tell you what tcpdump was, which of the eleven copies was current, or whether it was what you actually wanted.

The one exception was Whatis, a separate companion database of short human-written descriptions of software packages. Whatis was hand-curated metadata, not derived from the listings, and it is the only part of Archie that described what something was rather than where it was. That division — a machine-generated index of locations plus a hand-written index of meanings — is a pattern that recurs throughout the next decade, and it is why the directory era happened at all.

How you actually used it

There was no browser, so access came in four flavours, and the list is a good snapshot of what using the internet meant in 1991.

• A local client. You ran the archie command on your own machine, which spoke the Prospero/Archie protocol to a server.
• Telnet. You opened an interactive session on a public Archie server and typed queries at its prompt, sharing the machine with everyone else doing the same.
• Email. You sent a message containing find <string> to an Archie server and the results came back as a reply. This sounds absurd now and was completely normal then: a great many people had an internet connection that carried mail and nothing else, and query-by-email was how they used every service.
• Web gateways, from 1993. The best known was ArchiePlex, an HTML form front end written by Martijn Koster — the same person who a year later built ALIWEB and wrote the robots exclusion standard. For most people who ever saw Archie, ArchiePlex was the interface, which is a small irony: the pre-web search engine reached its widest audience through the web.

Searching Archie was therefore a skill, and the skill was guessing filenames. You had to know, or correctly infer, what a thing was called before you could find it. That is the exact inverse of how search works now, where you describe what you want in your own words and the engine reconciles that against how documents describe themselves.

Veronica and Jughead — the same idea for Gopher

Gopher, developed at the University of Minnesota, arrived in 1991 and briefly looked like the future. It organised internet resources into nested menus — you moved through a hierarchy of typed items until you reached a document, an image or a file. It was structured, fast and predictable in a way the early web was not, and for a couple of years serious institutions preferred it.

Gopher had exactly the same discovery problem as FTP, and it got exactly the same solution. Veronica and Jughead, which appeared between 1991 and 1993, searched Gopher menu titles — the human-written labels on menu entries — and returned the ones that matched. Veronica indexed across many Gopher servers; Jughead worked over a single server or a defined subset.

The naming is worth clearing up because it is a persistent misconception. Archie is not named after the comic-book character. Emtage has said the name is the word archive with the v removed. Veronica and Jughead, arriving afterwards, leaned into the joke deliberately — and their retrospective punning is why people assume Archie started it.

Structurally, Veronica and Jughead searched exactly what Archie searched: a short human-written label attached to a resource, not the resource. A Gopher menu title is a slightly richer object than a filename, because a human chose it to describe something, but it is still a label. Neither tool could find a document because of a phrase inside it.

WHOIS, and what does not count

WHOIS dates from 1982 and gets floated periodically as the true first search engine. It is not one. WHOIS is a lookup protocol for registration records: you query it with a domain name or a network block and it returns the administrative record for that registration. It does not search across content, it does not search across files, and it does not rank anything — it resolves an identifier to a record. Wikipedia's timeline of web search engines does not list it as an entry, and it should not be repeated as an earlier claimant without a stronger source than currently exists.

The general test is useful for judging any of these claims. A search engine takes a description of what you want and returns candidates ranked or filtered from a corpus it has indexed. A lookup service takes an identifier and returns the one record attached to it. Archie is a search engine over a corpus of filenames; WHOIS is a lookup. The distinction is not pedantry — it is what separates the tools in this article from the dozens of other network utilities of the same era.

Why the pre-web engines could not survive

They were displaced within about four years, and the reason is not that they were badly built. It is that the web changed the shape of the content underneath them, in three ways at once.

The web made documents discoverable. Hyperlinks meant a program could find a new page by reading an old one, so an index could grow on its own. Archie's list of FTP hosts had to be maintained by hand and could only ever know about servers someone had told it about. This is the difference between a harvest and a crawl, and it is the difference between an index that scales and one that does not.

The web made content indexable. Web pages are text, and text can be indexed by the words in it. Once WebCrawler demonstrated on 21 April 1994 that you could store the full text of every page and query it, the entire premise of guessing filenames became obsolete. You no longer had to know what a thing was called; you could describe it.

The web made the audience general. Archie's interface assumed you knew regular expressions or were comfortable composing an email to a server. That is fine for a network of a few hundred thousand technically expert hosts and useless for a consumer product.

Archie's development ceased in the late 1990s; version 3.5 in 1996 was the last release. Its last legacy production server, at the University of Warsaw, ran until 2023. In May 2024 the computer-history preservation project The Serial Port recovered the source from that Polish machine and launched a working public instance as an exhibit — but as of 19 August 2026 that instance is offline, its operators reporting that the machine running it is down. Anyone telling you Archie is back needs to check the date on the claim.

What the pre-web era left behind

Two inheritances outlived the tools themselves, and both are still visible.

The first is ALIWEB, whose name — Archie-Like Indexing in the WEB — is a direct statement of descent. Martijn Koster built it to bring Archie's model to the web: site owners would write a structured index file describing their own resources, register it once, and ALIWEB would harvest and merge those files. It is Archie's scheduled harvest of authoritative metadata, ported to a network where that model no longer worked, because the number of servers had grown past the point where anyone could be relied on to describe themselves. ALIWEB's failure is the clearest early demonstration that web-scale discovery has to be automatic and has to rest on evidence the publisher does not control.

The second is the idea of a separate human layer. Archie had Whatis for the same reason the mid-1990s had Yahoo's directory and the 2000s had DMOZ: a machine-built index of locations does not tell you what anything means, and for a long time the answer was to have people write the meanings down. That answer lasted until indexes got large enough and ranking got good enough to infer meaning from the corpus itself.

None of these tools is usable today, and none should be presented as one. Their value is that they make the definitional question precise. When someone says Archie was the first search engine, they are right, and the word doing the work is internet. When someone says it was the first web search engine, they are wrong, and no qualifier rescues it — the web did not exist yet.

Frequently asked questions

Sources

• en.wikipedia.org/wiki/Archie_(search_engine)
• en.wikipedia.org/wiki/Timeline_of_web_search_engines
• 200.mcgill.ca/history/creation-of-the-first-internet-search-engine/
• internethalloffame.org/inductee/alan-emtage/
• historyofinformation.com/detail.php?id=101
• en.wikipedia.org/wiki/ALIWEB
• archie.serialport.org/
• en.wikipedia.org/wiki/WebCrawler