How search engine crawling works

The fetch loop, and the queue that drives it

Crawling is the first of the three jobs a search engine does, and mechanically it is the simplest. A crawler holds a list of URLs it intends to fetch — conventionally called the frontier — takes one off, makes an ordinary HTTP request for it, stores the response, extracts every link in the returned HTML, and adds any URL it has not seen before back onto the frontier. Then it does it again. Forever.

Everything difficult about crawling is a consequence of running that loop billions of times against servers that did not consent to it in advance and cannot be assumed to cope. The loop itself is a first-year exercise. The scheduler around it is the hard part: deciding which of several hundred billion known URLs to fetch next, how often to come back, how fast to hit any one host, and which URLs to stop fetching altogether.

The crawler identifies itself on every request through the HTTP User-Agent header. Googlebot sends a string ending (compatible; Googlebot/2.1; +http://www.google.com/bot.html); bingbot sends the equivalent for bingbot/2.0. The pattern — a token, a version, and a URL pointing at documentation — is a convention rather than a requirement, but well-behaved crawlers follow it, and Marginalia takes it to its logical conclusion by using the bare string search.marginalia.nu, an address an administrator can simply visit to find out what hit their logs.

Where the URLs come from

A crawler cannot fetch a page it has never heard of, so URL discovery is the constraint that determines how much of the web an engine can see at all. There are five routes in, and most engines use several.

Links. The original and still the dominant mechanism. Every crawled page yields its outbound links, so the crawl expands outward through the web's own link graph from a set of seed URLs. A page nothing links to, on a site nothing links to, is invisible to this process.

Sitemaps and feeds. An XML sitemap is a machine-readable list of URLs a site publishes for crawlers. Marginalia supplements its main crawl with daily RSS polling, which lets it catch new posts on feed-publishing sites between full crawls that are otherwise eight to ten weeks apart.

Push notification. IndexNow, launched in October 2021 by Microsoft and Yandex, inverts the model: instead of waiting to be crawled, a site pings participating engines when a URL changes. Seznam adopted it in August 2022. It is a notable signal of index independence — only an engine that runs its own index has any use for change notifications.

Direct submission. Most independent engines run a submission route. Seznam operates an Add Page form and a submission API at reporter.seznam.cz; Naver has Search Advisor; Baidu has its Search Resource Platform. Marginalia's crawl is partly submission-driven by design, which is one reason its index is shaped differently from an engine that follows links alone.

Browser telemetry. Brave's Web Discovery Project is an opt-in system in the Brave browser contributing anonymised records of pages users visit, used to inform which URLs are worth crawling. It is not a substitute for crawling — the crawler still fetches — but it gives a small operator a signal about what exists that a pure link crawl would take far longer to find.

Politeness: the limits a crawler puts on itself

An unthrottled crawler is indistinguishable from a denial-of-service attack. Every credible crawler therefore rate-limits itself per host, and the good ones publish the number. Mojeek states that MojeekBot makes no more than one page request per second to any given site. Naver's Yeti and You.com's YouBot both honour the non-standard Crawl-delay directive in robots.txt, which asks for a minimum gap between requests. Baidu does not: it historically ignored Crawl-delay and directs publishers to set crawl frequency through its webmaster platform instead — and there is a long practitioner record of Baiduspider crawling aggressively and honouring new exclusions slowly.

Rate is only one axis. A crawler also has to handle server errors sensibly, back off when a host slows down, avoid fetching the same content through a hundred different URLs, respect a site's exclusion rules, and stop crawling regions that generate infinite URLs — calendars and faceted filters produce unbounded URL spaces that will consume a crawl budget forever.

The most concrete measure of whether an operator takes this seriously is documentation. Mojeek publishes crawl rate, robots.txt handling and a machine-readable IP list at mojeek.com/mojeekbot.json. Marginalia publishes its IP range, refresh cadence, blocking instructions and a human contact address for complaints. Seznam publishes a JSON IP range list and a reverse-DNS pattern. That is what an operator does when it actually runs a crawler and expects to be held responsible for it.

Scheduling and recrawl: why freshness costs money

The web changes constantly and a crawl is a snapshot, so every engine has to decide how often to go back. Recrawl frequency is where crawl budget is actually spent, and it is the single clearest difference between a large engine and a small one.

Big engines run tiered crawls. Yandex documents a "main" robot and a separate fast robot, the latter reserved for frequently updating sources such as news sites. Cốc Cốc runs a coccocbot-fast alongside its general web crawler for the same reason. The economics are straightforward: recrawling a news homepage every few minutes and an unchanged reference page every few months costs far less than treating them alike, and produces a fresher index.

Small engines cannot buy their way out of this. Marginalia runs a major index refresh on an eight-to-ten-week cycle, with crawl data given a stated shelf life of roughly two to three months. That makes it structurally poor at news and anything time-sensitive — which is an architectural consequence of running a web-scale crawler on a hobbyist's hardware budget, not a defect in the software. Mojeek's freshness lags Google's for the same reason: a smaller crawl budget cannot revisit news-velocity sites at the same cadence.

This is also why index size figures need context. Mojeek passed one billion pages in 2015, 4 billion in June 2021 and reports 9 billion as of 2025 — real numbers for a genuinely independent crawl, and one to two orders of magnitude below Google's, which is generally understood to be in the hundreds of billions. Both facts matter, and printing either alone misleads.

Rendering: the crawl that has to run a browser

A crawler that only reads the HTML the server sends will miss content a page builds in the browser with JavaScript. Solving that means running a real rendering engine over fetched pages, which costs orders of magnitude more CPU per page than parsing HTML.

The engines that do it say so. Google renders as standard and indexes mobile-first, so Googlebot Smartphone does most of the crawling. Baidu operates a distinct agent, Baiduspider-render/2.0, specifically to fetch the JavaScript-dependent resources a page needs. Microsoft documents MicrosoftPreview for page snapshot rendering alongside bingbot.

Smaller engines mostly do not render, and it shows in what they can see. Marginalia goes further and treats heavy JavaScript as a negative ranking signal rather than a rendering problem to solve — a page that requires a browser to say anything is, under its editorial model, less likely to be the kind of text it wants to surface. That is an unusually explicit statement of a trade-off every crawler operator makes implicitly.

Crawler fleets, not single crawlers

"Googlebot" is shorthand. Every major engine runs a family of agents with different jobs, different rules and different tokens, and blocking one does not block another.

• Baidu runs Baiduspider plus Baiduspider-image, -video, -news, -favo, -cpro, -ads and the rendering agent.
• Cốc Cốc documents coccocbot, coccocbot-web, coccocbot-image, coccocbot-fast, coccocbot-ads and coccocbot-shopping, each with a published user-agent string.
• Yandex publishes a table of roughly forty named robots — YandexImages, YandexMobileBot, YandexMetrika, YandexMarket, YandexVideo and others — and records for each whether it obeys general robots.txt directives. Some are documented as not doing so.
• Google separates Googlebot from GoogleOther, a generic fetcher used by various product teams, and from Google-Extended, which is a robots.txt token with no user-agent of its own that governs whether content is used to train and ground Gemini models.
• Microsoft runs adidxbot for ad landing-page verification alongside bingbot.
• OpenAI separates OAI-SearchBot, which governs appearance in ChatGPT search answers, from GPTBot, which governs model training, from ChatGPT-User, which fetches a page because a user asked for it.

The ads and shopping crawlers are worth noticing because they reveal the business underneath: coccocbot-ads and adidxbot exist to check that advertisers' landing pages are what they claim to be.

Proving a crawler is who it says it is

A user-agent string is just a header. Anyone can send it, and Baiduspider is among the most-spoofed user agents on the web — traffic scraping a site while wearing a search engine's name is a routine problem, and Yandex's own documentation warns explicitly that "some robots can disguise themselves as Yandex robots."

Two verification schemes are in general use. The first is a DNS round trip: take the requesting IP, look up its reverse-DNS hostname, confirm the hostname ends in a domain the engine controls, then forward-resolve that hostname and confirm it returns the same IP. Yandex documents this four-step method against yandex.ru, yandex.net and yandex.com; Seznam publishes the PTR pattern fulltextrobot-77-75-77-xxx.seznam.cz; Baidu verifies against *.baidu.com; You.com uses youbot-{ip-octets}.search.you.com. The second is a published IP list, served as JSON so it can be checked automatically — Mojeek, Marginalia, Seznam and Perplexity all do this.

The newest approach is cryptographic. You.com's crawler signs its requests using Cloudflare's Web Bot Auth scheme, so a server can verify the signature rather than trusting DNS. That closes the gap properly, and it is the direction the problem is moving — because as AI crawling has grown, the value of impersonating a well-regarded crawler has grown with it.

What crawlers cannot reach

A crawl only sees what an anonymous HTTP request returns. Anything behind a login, a paywall, a form submission or a strict bot-detection rule is outside it, and the volume of that material has grown considerably.

The most consequential example is contractual rather than technical. In 2024 Reddit blocked non-Google crawlers, cutting Mojeek and other independent engines off from a large body of discussion content while Google retained access under a paid arrangement. Nothing about Mojeek's crawler failed; the door was closed. That single decision illustrates the structural problem better than any technical limit does — index consolidation is now maintained by licensing, not by engineering, and a small engine cannot negotiate its way past it.

Infrastructure providers have also become gatekeepers, maintaining verified-bot directories that determine which crawlers pass through managed rules and which are challenged. Being catalogued as a legitimate crawler — Cloudflare's directory lists Bravebot, Marginalia's agent, Baiduspider and OAI-SearchBot among others — is now a practical prerequisite for crawling much of the web, and removal from such a list is a real event: Cloudflare de-listed Perplexity in August 2025 after publishing research alleging it used undeclared crawlers with generic browser user-agents to fetch pages that had blocked PerplexityBot. Perplexity disputed the characterisation, arguing the traffic was user-initiated.

Frequently asked questions

Sources

• mojeek.com/bot.html
• about.marginalia-search.com/article/crawler/
• help.coccoc.com/en/search-engine/coccoc-robots
• yandex.com/support/webmaster/robot-workings/check-yandex-robots.html
• developers.google.com/search/docs/crawling-indexing/google-common-crawlers
• o-seznam.cz/napoveda/vyhledavani/en/seznambot-crawler/
• you.com/docs/youbot
• indexnow.org/