What ALIWEB was, and where its results came from
ALIWEB was an early searchable directory of web resources assembled entirely out of descriptions that site owners wrote about themselves. A webmaster created a small structured text file on their own server — conventionally site.idx — listing each resource with a title, a description, keywords and a URI, then registered the location of that file with ALIWEB through a form. ALIWEB fetched the registered index files on a schedule, merged them, and let people search the result.
What you got back was therefore a list of self-described sites, not an index of page text. Nothing in ALIWEB's database had been read by a machine that visited the page in question. There was no crawler, no full-text retrieval, and no independent evidence of any kind about the pages listed — only publishers' claims about their own work.
The name is the design brief, not a coincidence: ALIWEB stands for Archie-Like Indexing in the WEB, per the title of Koster's own 1994 conference paper. Most secondary sources render it “Archie-Like Indexing for the Web”, but the author's wording is the one to prefer. The point of the name was that the web now needed the kind of catalogue Archie had already provided for public FTP servers.
It was not the first internet search engine
This is the most common thing said about ALIWEB and it is false. It is false in Wikipedia's own opening sentence, which reads: “ALIWEB (Archie-Like Indexing for the Web) was the first internet search engine.” A large volume of downstream writing has copied that claim.
Archie went live on 10 September 1990, more than three years before ALIWEB was announced. Archie was a searchable catalogue of the filenames held on public anonymous-FTP servers, and it is the strongest claimant to the title of first internet search engine. ALIWEB is named after it. The error is self-refuting: a service whose name announces that it does for the web what Archie did for FTP cannot also be the first thing of its kind.
A narrower claim about ALIWEB is defensible. It was among the earliest web search services, and it predates every crawler-based engine that went on to matter. That is a real place in history, and it does not require the false version.
How the submission model worked
From Koster's 1994 paper, the mechanism ran in four steps:
- A site administrator writes an index file, normally site.idx at the server root, using IAFA templates — Internet Anonymous FTP Archives templates, an attribute/value format modelled on RFC 822 mail headers. The fields include Template-Type, Title, Description, Keywords, URI and organisation details.
- The administrator registers the URL of that index file with ALIWEB once, through a web form.
- ALIWEB retrieves the registered index files on a schedule, parses and validates them, and merges them into a single searchable database.
- Users search that merged database of human-written descriptions.
This was a deliberate rejection of robots, not an oversight. Koster's paper argues that crawlers imposed “considerable network overhead”, could overload servers, retrieved large volumes of irrelevant documents, and destroyed the structure of the information they gathered — behaviour he described as unacceptable to many server administrators. Read in 1994, that is a reasonable position. Read now, it is the losing side of the argument that defined web search.
Why the model failed
Three reasons, and only the first is the one usually given.
It required unpaid work from every site owner, forever. Coverage was capped by the number of administrators who bothered to write an IAFA template and keep it current. The overwhelming majority never heard of ALIWEB; of those who did, few maintained the file. Koster himself noted that some people registered without actually providing an index file at all. A crawler-based rival got complete coverage of a site for free, with the site owner doing nothing and knowing nothing.
Self-description is unverifiable and gameable. Every entry was whatever the publisher claimed about their own pages. There was no page text to check the claim against, no links, and no ranking signal of any kind. The same weakness later destroyed the value of the HTML keywords meta tag, for precisely the same reason: metadata the publisher controls and nobody verifies converges on whatever gets the most traffic.
The retrieval itself was weak. Per Wikipedia, the original database did not search the whole database for a query — it scanned from the beginning until it ran out of results or hit a limit. A later attempt to fix this by weighting results and searching the full database came too late to matter.
Nexor's own retrospective offers a fourth and softer explanation, worth quoting because it comes from the operator: as spidering services proliferated, “Aliweb became one of many such search engines. Their individual coverage was limited”, and users had to work out which engine to use for which search. That describes the mid-1990s accurately but understates the problem. The crawler-based engines' coverage grew automatically; ALIWEB's could not grow at all without volunteers.
By the time WebCrawler arrived in April 1994 with full-text search, followed by Lycos, Infoseek, Excite and AltaVista over 1994–95, a directory of a few thousand voluntary self-descriptions had no answer to any of them.
Who built it, and the robots.txt irony
ALIWEB was the work of Martijn Koster, then at Nexor Ltd in Nottingham, in the United Kingdom. There was no business model: it was a research and demonstration project connected to Nexor's work on internet resource discovery, with no advertising and no subscription, and no evidence has been located that it ever generated revenue.
Koster is also the author of the Robots Exclusion Standard — robots.txt — which he wrote in 1994, the same year ALIWEB was presented, and from the same conviction about crawler behaviour. The irony is exact: the standard he wrote is what made large-scale crawling socially acceptable to server administrators, and large-scale crawling is what made his own engine obsolete. Three decades on, robots.txt is honoured on essentially every web server, and ALIWEB survives only in archives.
Koster had form in this area already. He also wrote ArchiePlex, the HTML form front end that put Archie in a browser — which is the other half of why ALIWEB is named the way it is.
Was it the first web search engine?
More defensible than the “first internet search engine” claim, but still contested, and the honest answer is that it depends on definitions and on which date you use for ALIWEB.
Nexor's own retrospective claims Aliweb is “acknowledged as the world's first search engine” and specifically criticises the BBC for “incorrectly citing WebCrawler”. But Nexor compares ALIWEB only to WebCrawler, of April 1994, and not to the earlier claimants:
- World Wide Web Wanderer — June 1993. The first known web robot; its index was called Wandex.
- W3Catalog — 2 September 1993. Described in Wikipedia's timeline as “the world's first web search engine”.
- JumpStation — began indexing 12 December 1993. The first to combine crawling, indexing and searching.
- WebCrawler — April 1994. The first to offer full-text search of the words inside pages.
ALIWEB's announcement in late 1993 sits among those dates; its public launch in May 1994 sits after all of them. Much of the ordering confusion comes from writers picking whichever of ALIWEB's dates suits their sentence. The defensible claim is the narrow one: ALIWEB was among the earliest web search services and predates every crawler-based engine that went on to matter.
Dates, and what happened to it
Three dates are all genuine, and any page that gives a single bare year invites contradiction:
- 1992 — Nexor's own account says Koster built it in this year.
- October/November 1993 — announced. Wikipedia's timeline gives October/November; the ALIWEB article says November.
- May 1994 — presented publicly at the First International Conference on the World Wide Web at CERN in Geneva, where the primary source paper appears in the proceedings. This is the date usually cited as the launch.
The ending is less documented than the beginning. No announced shutdown date has been located, in Wikipedia or in Nexor's retrospective. Wikipedia lists the status as inactive and records an archived snapshot dated 18 June 1997, which is the last firm evidence of the service. As of 19 August 2026 the domain aliweb.com resolves in DNS, but it is unrelated to Koster's project.
Why it still matters
ALIWEB was intellectually coherent and practically doomed. Its diagnosis of crawler behaviour was correct enough that Koster's other 1994 contribution is still enforced everywhere. Its remedy — ask the whole world to describe itself accurately and voluntarily — is the kind of design that works at five hundred servers and collapses at fifty thousand.
Its failure is the clearest early demonstration that web-scale discovery has to be automatic, and has to rest on evidence the publisher does not control. That lesson did not stop the model recurring: meta keywords, directory submissions, XML sitemaps and IndexNow are all descendants of the same idea, and each has been treated by search engines with the same institutional scepticism ALIWEB earned for it.
Two further corrections are worth stating plainly. ALIWEB never crawled the web — that was the entire point of it, not a limitation. And it did not fail because it was too early: contemporaries that crawled survived years longer. It failed because its data collection did not scale and its data could not be trusted.
Frequently asked questions
Was ALIWEB the first internet search engine?
No. Archie went live on 10 September 1990, more than three years before ALIWEB was announced in late 1993. Wikipedia's article on ALIWEB opens by calling it the first internet search engine, and that claim is incorrect and widely copied. ALIWEB's own name — Archie-Like Indexing in the WEB — refers to Archie, which makes the error self-refuting.
What does ALIWEB stand for?
“Archie-Like Indexing in the WEB”, per the title of Martijn Koster's own 1994 conference paper. Most secondary sources render it “Archie-Like Indexing for the Web”. Either way, the name is a reference to Archie, the 1990 search service that catalogued filenames on public FTP servers, and it describes the intent: do for web resources what Archie did for FTP archives.
Did ALIWEB have a crawler?
No, and that was deliberate. Site owners wrote a structured index file, conventionally site.idx, describing their own resources, then registered its URL with ALIWEB once. ALIWEB fetched the registered files on a schedule and merged them into one searchable database. Koster's paper argues explicitly that crawlers overloaded servers, wasted bandwidth and retrieved irrelevant documents.
Why did ALIWEB fail?
Three reasons. It depended on unpaid work from every site owner forever, so coverage never grew on its own. Its data was pure self-description, which is unverifiable and gameable. And its retrieval was weak — the original database scanned from the beginning until it hit a limit rather than searching everything. Crawler-based engines got complete coverage for free.
When was ALIWEB launched?
Three dates are all real and sources differ on which to use. Nexor says Koster built it in 1992. It was announced in October or November 1993. It was presented publicly at the First International Conference on the World Wide Web at CERN in May 1994, which is the date usually cited as its launch and the date of the primary paper.
Who created ALIWEB?
Martijn Koster, while working at Nexor Ltd in Nottingham, United Kingdom. Koster also wrote the Robots Exclusion Standard — robots.txt — in 1994, and ArchiePlex, the HTML form front end for Archie. The robots standard he authored is what made large-scale crawling acceptable to server administrators, and large-scale crawling is what made his own engine obsolete.
Is ALIWEB still online?
No. No announced shutdown date has been found in Wikipedia or in Nexor's own retrospective, so there is no clean end date to cite. Wikipedia lists the status as inactive and records an archived snapshot dated 18 June 1997, which is the last firm evidence of the service running. The domain aliweb.com resolves today but is unrelated to the original project.
What was the first web search engine, if not ALIWEB?
It depends on the qualifier. The World Wide Web Wanderer, of June 1993, was the first known web robot. W3Catalog, of 2 September 1993, is called the world's first web search engine in Wikipedia's timeline. JumpStation, from 12 December 1993, was the first to combine crawling, indexing and searching. WebCrawler, of April 1994, was the first with full-text search.