What is a web crawler?

One program, four names

A web crawler is a program that fetches a page over HTTP, extracts the links inside it, and goes on to fetch those. That is the whole definition. Everything else — search indexes, web archives, AI training corpora, broken-link reports — is built on top of that one behaviour.

It is called several things, and the names are interchangeable rather than technical. Crawler is the plain description. Spider is the metaphor from crawling a web, and was the dominant term in the 1990s. Robot is the oldest usage: the earliest such program, Matthew Gray's World Wide Web Wanderer of June 1993, was described as a robot, which is why the exclusion standard that arrived a year later is stored in a file called robots.txt and why the whole class of them are still "robots" in that context. Bot is the modern shorthand and is broader — it covers plenty of automated traffic that is not crawling at all.

Nothing distinguishes a crawler from a browser at the protocol level. Both send an HTTP request and receive a response. A server can only tell them apart by what the request says about itself, chiefly in the User-Agent header, and by behaviour — a browser fetches one page and the images on it, a crawler fetches thousands of unrelated pages steadily for hours.

What is actually inside one

Structurally a crawler is four components and a queue.

• The frontier is the queue of URLs waiting to be fetched, usually prioritised rather than first-in-first-out, and partitioned by host so that one site's URLs can be spaced out.
• The fetcher makes the HTTP requests. AltaVista's crawler Scooter, written by Louis Monier at Digital Equipment Corporation's Western Research Laboratory, is the one usually credited with getting this right first: it was multi-threaded, keeping large numbers of fetches in flight at once instead of crawling serially. That concurrency, not any ranking innovation, is why AltaVista could build an index an order of magnitude larger than its contemporaries in 1995.
• The parser pulls the links and the text out of what came back.
• The store writes the fetched content somewhere for the indexer to process later.

Around all of that sits a scheduler enforcing politeness rules — how fast to hit a host, when to come back, what the site's exclusion rules permit. A crawler without a scheduler is a load generator. That is the difference between the exercise and the system, and it is where every serious operator spends its engineering time.

The crawlers you will actually see

Search crawlers announce themselves, and the documented ones are a small enough set to list. The pattern of the user-agent string is a convention: a name token, a version, and a URL pointing at documentation.

• Googlebot — Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36, with a Smartphone variant that does most of the crawling, since Google indexes mobile-first.
• bingbot — (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm), in desktop and mobile forms, current since Microsoft's April 2022 user-agent change.
• MojeekBot — Mozilla/5.0 (compatible; MojeekBot/0.11; +https://www.mojeek.com/bot.html). Mojeek publishes crawl rate, robots.txt handling and an IP list at mojeek.com/mojeekbot.json.
• Bravebot — (compatible; Bravebot/1.0; +https://search.brave.com/help/brave-search-crawler). Notable because Brave's earlier position was that its crawler deliberately did not advertise a differentiated user-agent, on the reasoning that it had to avoid discrimination from sites that allow only Google. Publishers who wanted to block it could not do so cleanly; the declared Bravebot token resolves that complaint, and older documentation describing the generic UA is out of date.
• search.marginalia.nu — the bare user-agent Marginalia uses, chosen because it is an address a server administrator can simply visit. It crawls from a published static range, 193.183.0.162–193.183.0.174.
• SeznamBot — (compatible; SeznamBot/4.0; ...), verifiable by both a published JSON IP list and reverse-DNS records of the form fulltextrobot-77-75-77-xxx.seznam.cz.
• coccocbot — Cốc Cốc runs six documented agents: coccocbot, -web, -image, -fast, -ads and -shopping.
• Yeti — Yeti/1.1 (Naver Corp.; http://help.naver.com/robots/), Naver's crawler, which honours robots.txt and Crawl-delay.
• Baiduspider, YandexBot, DuckDuckBot, PerplexityBot, YouBot and OAI-SearchBot all follow the same shape.

Two historical names still appear in old documentation: Slurp, Yahoo's crawler from the era when Yahoo ran its own index, and Scooter, AltaVista's. Neither backs a live general web index today.

Crawlers that are not search engines

A growing share of crawl traffic has nothing to do with building a search index, and the categories have different implications for the sites being fetched.

AI training crawlers collect text to train models. OpenAI's GPTBot is the clearest example, and it is deliberately separate from OAI-SearchBot, which governs whether a page can appear in ChatGPT search answers. Google's Google-Extended is a robots.txt token with no user-agent of its own, controlling whether content is used to train and ground Gemini models — and Google states it does not affect inclusion in Search. That separation lets a publisher say "index me, do not train on me", and it is the single most commonly inverted fact in this area: blocking GPTBot does not remove a site from ChatGPT's search answers, and blocking OAI-SearchBot does.

User-initiated fetchers retrieve a specific page because a person asked an assistant to look at it. ChatGPT-User and Perplexity-User are both in this class, and both operators state that robots.txt rules may not apply to them, on the reasoning that the request is not a crawl. That reasoning is contested and is the origin of much of the current conflict between AI companies and publishers.

Ad verification crawlers check that advertisers' landing pages are what they claim. Microsoft's adidxbot and Cốc Cốc's coccocbot-ads exist for this and nothing else.

Archival crawlers fetch pages to preserve rather than serve them, monitoring tools crawl on behalf of customers, and security scanners crawl looking for vulnerabilities. A substantial volume of unnamed automated traffic is straightforwardly hostile. All of it looks similar in a log file, which is why identification matters.

Verifying that a crawler is what it claims

The user-agent header is self-reported and trivially forged. Baiduspider is among the most impersonated strings on the web, and Yandex's own documentation warns plainly that "some robots can disguise themselves as Yandex robots" by sending the relevant header. Any conclusion drawn from the header alone is unreliable.

The established check is a DNS round trip. Take the IP address the request actually came from, look up its reverse-DNS hostname, confirm that hostname sits under a domain the engine controls, then forward-resolve the hostname and confirm it resolves back to the same IP. Yandex documents this as a four-step process against yandex.ru, yandex.net and yandex.com. Baidu documents it against *.baidu.com and *.baidu.jp. You.com uses the pattern youbot-{ip-octets}.search.you.com. Mojeek supports it against mojeek.com.

The alternative is a published IP list, generally served as JSON so it can be consumed automatically: mojeek.com/mojeekbot.json, search.marginalia.nu/crawler-ips.txt, Seznam's IP range list, Perplexity's perplexitybot.json and perplexity-user.json. Mojeek notes that its list can change at any time, which is the honest caveat on the method.

The newest approach is cryptographic signing. You.com's crawler signs its requests under Cloudflare's Web Bot Auth scheme, so a server verifies a signature rather than trusting DNS at all. That is the only method in this list that a forger cannot defeat by controlling network position, and it is where the problem is heading, because as the commercial value of crawling has risen so has the incentive to impersonate a crawler that servers already trust.

What a well-behaved crawler does

There is no enforcement body for crawler conduct. What exists is a set of conventions and a reasonably clear line between operators who follow them and operators who do not.

The strong end of the range is well documented. Mojeek publishes its user-agent, states a crawl rate of no more than one request per second per site, documents how it matches robots.txt records, supports noindex, nocache and nofollow, and offers two independent verification methods. Marginalia publishes its user-agent, a static IP range, a machine-readable IP list, its refresh cadence, blocking instructions and a named human contact address for complaints. You.com documents full robots.txt compliance including per-agent rules and Crawl-delay, with a thirty-minute robots.txt cache, plus a published crawl range, reverse-DNS verification and cryptographic request signing.

The weak end is also documented, mostly by the operators themselves. Perplexity's own documentation states that Perplexity-User generally ignores robots.txt. In August 2025 Cloudflare published research alleging Perplexity additionally used undeclared crawlers with generic browser user-agents and rotating networks to fetch content from sites that had explicitly blocked PerplexityBot, and de-listed it from its verified-bot programme; Perplexity disputed the characterisation, arguing the traffic was user-initiated. Baidu presents a third case: Baiduspider states it observes robots.txt, but there is a long practitioner record of aggressive crawling and slow honouring of new exclusion rules, and it has historically ignored Crawl-delay entirely, directing publishers to its webmaster platform instead.

The pattern worth noticing is that crawler documentation quality is a reasonable proxy for whether an operator genuinely runs a crawler at all. Gibiru has marketed itself as an independent search engine for seventeen years and has never named a crawler, published a user-agent, or documented a crawl scope. An operator that actually runs a crawler wants server administrators to be able to work with it.

The first crawlers, and what they proved

Crawling arrived almost immediately after the web did, and the arguments about it arrived with it.

Matthew Gray's World Wide Web Wanderer ran in June 1993, initially to measure the web's size, and its index was called Wandex. JumpStation began indexing on 12 December 1993, using a crawler but indexing only titles and headings. WebCrawler, launched by Brian Pinkerton at the University of Washington on 21 April 1994 with an index of just over four thousand sites, was the first to index the full text of the pages it fetched — earlier tools indexed filenames, titles or human-written summaries. It served its millionth query on 14 November 1994. Every general web search engine since is built on Pinkerton's model.

The counter-argument came at the same time from the same community. Martijn Koster's ALIWEB, presented at the first World Wide Web conference at CERN in May 1994, deliberately refused to crawl: site owners wrote their own index files and registered them. Koster's paper argued that robots imposed considerable network overhead, could overload servers, and retrieved large volumes of irrelevant documents. He lost that argument on the merits — a directory of a few thousand voluntary self-descriptions had no answer to a crawler that covered a site for free — but he won the procedural one, authoring the Robots Exclusion Standard the same year.

The name that survives least accurately is WebCrawler's own. It has not run a crawler since 2001, when its then-owner switched it to Excite's database; today it is a System1-owned front end over third-party feeds. The site named after the first full-text web crawler crawls nothing.

Frequently asked questions

Sources

• mojeek.com/bot.html
• about.marginalia-search.com/article/crawler/
• search.brave.com/help/brave-search-crawler
• you.com/docs/youbot
• docs.perplexity.ai/guides/bots
• developers.openai.com/api/docs/bots
• yandex.com/support/webmaster/robot-workings/check-yandex-robots.html
• en.wikipedia.org/wiki/WebCrawler