The directory era of web search

A directory is not a search engine

For roughly a decade, the most useful way to find things on the web was not to search it but to browse a catalogue people had built by hand. A web directory listed sites, not pages, arranged into a subject hierarchy, each with a short description written by an editor. You did not type a query and get ranked matches; you walked down a tree — Recreation, then Outdoors, then Climbing — and read what was on the shelf.

The distinction matters because the two get conflated constantly, including by people who lived through it. A search engine runs a crawler, stores an index of page content and ranks matches algorithmically. A directory has none of that: it has editors, a taxonomy and a queue of submissions. DMOZ, the largest directory ever built, never crawled anything and never indexed a word of page text — and it supplied data to search engines rather than buying results from them.

Yahoo — the directory that became a portal

Yahoo began in 1994 as Jerry and David's Guide to the World Wide Web, a hand-built list compiled by Jerry Yang and David Filo at Stanford, and its search box searched that list. This is why calling Yahoo an early search engine is misleading: for its first several years its product was editorial judgement about which sites deserved a listing, and its results were as good as its editors and no better.

Two things then happened that define the era. Yahoo needed results for queries its directory could not answer, so it bought them — from Open Text, then AltaVista, then Inktomi, then Google from 2000 to 2004. For four years the web results on Yahoo were Google's. And the directory itself became a business: commercial submission moved to a paid review model, eventually at 299 dollars, or 600 for adult sites, per Wikipedia's account. Paying bought a review, not a listing — a distinction thin enough in practice to damage the perception that directory listings were editorial at all.

In October 2002 Yahoo switched its main results to crawler-based listings under Yahoo Search, which is the moment the directory stopped being the product and became a section of a portal. It survived in that reduced form for another twelve years: Yahoo announced its closure on 26 September 2014 and shut it on 31 December 2014. Yahoo did not operate a web crawler of its own until February 2004, after it had bought Inktomi, AltaVista and AllTheWeb, and it gave that index up again after the Microsoft search alliance of July 2009. Today Yahoo's web results are Bing's.

Magellan — reviews, star ratings and a green light

Magellan, run by The McKinley Group of Sausalito, California — founded by sisters Christine and Isabel Maxwell — tried hardest of the directories to add editorial value rather than just editorial coverage. Its editors did not merely list sites; they reviewed them, writing a description and assigning a rating from one to four stars. A green-light icon marked sites judged suitable for all ages — an entirely human form of family filtering, years before automated safe search existed. Magellan carried two databases, the reviewed collection and a much larger unreviewed one, and users could set a minimum star rating to restrict results to sites an editor had actually vetted.

Its dates need care. The conventional launch year is 1995, with the company founded around 1993, but no primary source fixing an exact date could be located and the widely circulated formulation traces to unreliable summaries rather than a citable original — treat 1995 as conventional and do not assert a month. Whether McKinley ran its own crawler for the unreviewed database is genuinely unestablished: no crawler documentation or user-agent has been found, and it is equally plausible that database was licensed or assembled from submissions.

Its commercial failure was mundane. The Virginian-Pilot reported in June 1996 that Magellan was late in selling advertising — it started in February 1996 while competitors had begun the previous autumn, and in a land grab measured in months, six months was fatal. Excite agreed to acquire McKinley on 28 June 1996 for about 18 million dollars in stock plus assumption of debt, kept the brand alive as a secondary front end for nearly five years, and shut Magellan down in May 2001.

LookSmart — wholesaling the catalogue

LookSmart shows what the directory business actually was. Founded in Melbourne, Australia in 1995 as Homebase by Evan Thornley and Tracey Ellery and renamed LookSmart on 28 October 1996, its product was a directory built by paid editors — but its business was not running a destination site. It was wholesaling the catalogue to portals, above all Microsoft's MSN Search, which displayed LookSmart's categorised listings for years.

That deal, struck in early 1999, is the whole company. Wikipedia's account gives roughly 30 million dollars up front plus about 5 million a year over five years, and by August 2003 Microsoft accounted for something like 64 to 70 per cent of LookSmart's revenue. When Microsoft announced on 6 October 2003 that it would not renew, the stock fell by more than half in a day and the company never recovered.

LookSmart ran the era's other characteristic experiments too. It bought Zeal, a volunteer-editor directory, in October 2000 and closed it on 28 March 2006; it converted flat-fee listings to cost-per-click in 2002, provoking a backlash and litigation from small site owners; and it bought WiseNut in March and April 2002, a genuine crawler-based engine with its own index and the only real search technology LookSmart ever owned. WiseNut never gained traction and was shut down in late September 2007. Calling LookSmart a search engine is a category error for most of its history: it was a directory, then briefly a directory with an engine attached, then an advertising network.

DMOZ — the volunteer directory that outlived them all

The Open Directory Project, universally called DMOZ after its home at directory.mozilla.org, launched on 5 June 1998 as GnuHoo, was renamed NewHoo after objections from the GNU project and Yahoo, and became the ODP when Netscape acquired it in October 1998. AOL inherited it in 1999; Verizon inherited it in 2015.

It was the largest human-edited catalogue of the web ever assembled, and built entirely by unpaid volunteers: editors applied for a category, decided what deserved a listing, wrote a one-line description and slotted it into the hierarchy. Critically, listings were free — the explicit contrast with the Yahoo Directory's paid review, and the reason anyone offering to sell you a DMOZ listing was selling something they did not control.

What made DMOZ structurally important was not the site but the weekly RDF dumps, which anyone could download and reuse — which is how DMOZ ended up underneath other people's products rather than competing with them. Google Directory was DMOZ's hierarchy and descriptions re-sorted by PageRank; Netscape Search, AOL Search, Alexa, Lycos and HotBot all carried ODP listings at various points; and engines used ODP descriptions as fallback snippets when a page had no usable meta description, which is why the NOODP robots meta tag existed. In 2011 the licence was quietly changed to Creative Commons Attribution, and that change is why the data legally survived the shutdown.

The decline is visible in DMOZ's own counts. It passed 1 million URLs in October 1999, overtook the Yahoo Directory around April 2000 and reached 5.17 million sites across 1.017 million categories by April 2013 — then fell to 3.996 million by 31 October 2015. A shrinking directory is one losing to link rot faster than volunteers can add to it. A catastrophic server failure between 20 October and 18 December 2006 stopped editing for nearly two months, and Google shut down Google Directory in July 2011, removing DMOZ's largest distribution channel. A notice in early March 2017 gave 14 March 2017 as the closure date; the site actually went to a single landing page on 17 March 2017. Both dates circulate, and the honest thing is to say so.

Why curation lost

The directories did not lose because their editors were bad. They lost to an arithmetic problem no amount of editorial quality could solve.

Human review does not scale, and the web grew faster every year. Crawling a site costs the owner nothing and the engine almost nothing, and coverage grows automatically. A directory's coverage costs an editor's attention per site, forever, and that attention is fixed. The contemporaneous guide that praised Magellan's reviews also warned it was probably not the best resource for uncommon topics — the fatal flaw of every directory, stated politely: editors cover what is popular, and the long tail is where search is most valuable.

Queues became the product. Submission backlogs of months to years in commercial DMOZ categories were routine and openly complained about, and the project's defence was that editors were volunteers and unaccountable to submitters by design. Allegations that editors favoured their own sites are well documented in contemporaneous forums, though systematic evidence of corruption is thin. Either way, an index whose freshness depends on a volunteer's continued interest in a category has an availability problem no algorithm has.

And paid placement corrupted the signal. Yahoo's paid review and LookSmart's paid inclusion meant the listing you saw was partly a function of who had paid for consideration. That is not fraud — both disclosed it — but it removed the one advantage a directory had over a crawler: the credibility of a disinterested human judgement.

Meanwhile the engines got better at the exact thing directories existed for. A directory answered the question what is worth reading on this subject. Once link-analysis ranking arrived, a crawler could infer that answer from the corpus itself, without anyone writing it down — the moment curation stopped being necessary rather than merely slow.

What survived

DMOZ's data outlived DMOZ, because of the 2011 relicensing. As of 19 August 2026:

• Curlie is the official successor, announced in September 2017 and run by former ODP editors. It kept the hierarchy and still accepts new listings. Its home page counters read 3,000,833 sites and 1,031,646 categories when checked on 19 August 2026 — well below DMOZ's 2013 peak, because Curlie has pruned heavily. These are live counters and will drift.
• ODP.org is a deliberately frozen snapshot built from the final DMOZ RDF file by former editors, stating explicitly that it is not adding new sites. It reports roughly 3.86 million listings — DMOZ's state at closure. It is the better citation for DMOZ as it was; Curlie is DMOZ as it continued.
• dmoztools.net, the mirror that a great many articles and Wikipedia references still point at, no longer resolves in DNS. Those links are dead. This is the most common stale fact about DMOZ on the web.
• The raw RDF dumps still circulate and are still used as a labelled taxonomy corpus in text-classification research — a quietly impressive afterlife for a volunteer project.

Every major directory of the era is now closed: Google Directory since 2011, the Yahoo Directory since 2014, DMOZ since 2017, LookSmart's directory and Zeal for longer still. Offers to submit a site to any of them are fraudulent, and directory-submission fraud using dead directory names has been a persistent problem for two decades. What the archives are genuinely good for is more interesting: they are the largest body of human-written, non-promotional descriptions of websites in existence, and if you want to know what the web looked like to informed people in 2005, they are a better artefact than any crawl of the same year.

Frequently asked questions

Sources

• en.wikipedia.org/wiki/DMOZ
• en.wikipedia.org/wiki/Yahoo!_Directory
• en.wikipedia.org/wiki/LookSmart
• en.wikipedia.org/wiki/Excite_(web_portal)
• scholar.lib.vt.edu/VA-news/VA-Pilot/issues/1996/vp960629/06290210.htm
• searchengineland.com/rip-dmoz-open-directory-project-closing-270291
• odp.org/about.html
• curlie.org/