Yahoo! and the curated directory
Yahoo! began in 1994 as "Jerry and David's Guide to the World Wide Web," a personal bookmark list maintained by Stanford graduate students Jerry Yang and David Filo. It quickly grew into a formally organized hierarchical web directory, with human editors reviewing and categorizing submitted sites into nested categories — Arts, Business, Recreation, and so on. Getting listed in the Yahoo! Directory was, for several years, one of the most valuable things a website could do, and an entire submission industry grew up around it.
The directory model had a real strength: because a human reviewed every listing, results were generally free of the spam and manipulation that would later plague purely algorithmic engines. Its weakness was equally fundamental — it could not scale. Editorial review that worked reasonably well for a web of a few thousand sites became untenable as the web grew into the millions, and directories increasingly could only cover a shrinking fraction of what existed.
The crawler-based competitors
Alongside Yahoo!'s directory, a wave of crawler-based engines built their own automated indexes, competing on coverage and relevance rather than editorial curation:
- Lycos (1994), spun out of Carnegie Mellon University research, was among the first to combine a large crawled index with basic relevance ranking.
- Excite (1995) marketed itself on "concept-based" search, an early attempt at understanding query meaning beyond literal keyword matches.
- AltaVista (1995), built by Digital Equipment Corporation, was for several years the technical leader of the field — the first engine to offer natural-language queries and one of the largest indexes of its time.
- Infoseek (1994) offered both directory browsing and crawler-based search, and briefly powered Netscape's default search results.
- Ask Jeeves (1996) differentiated itself with a natural-language question-answering interface, letting users type full questions rather than keyword strings — a UX idea that would resurface, in far more capable form, in today's AI search assistants.
Why keyword-based ranking broke down
Most of these crawler-based engines ranked pages primarily by keyword frequency and basic on-page text matching: a page mentioning a search term more often, or more prominently, ranked higher. This was straightforward to compute at scale, but it was also trivial to manipulate. Keyword stuffing — repeating a term far beyond what natural writing would produce, often in invisible or hidden text — became a common tactic, and search results across the industry grew increasingly unreliable as a result.