Robots.txt and crawler control

A text file at the root of a site, and a request

robots.txt is a plain text file served from one fixed location — the root of a host, at /robots.txt — that tells automated clients which parts of that host they are asked not to fetch. It is the oldest surviving web convention governing machine behaviour rather than human behaviour, and its most important property belongs in the first sentence of any honest description of it: it is a request.

Nothing enforces it. A crawler that ignores robots.txt receives exactly the same responses as one that reads it. The file is not access control, authenticates nothing, and cannot prevent a fetch. It works to the extent that operators choose to make it work — which, for the search crawlers documented on this site, is very largely, and for some newer categories of crawler, much less so.

The mechanism is deliberately crude. A client fetches /robots.txt before crawling a host, parses it, finds the group of rules addressed to itself, and applies them for some cache period — You.com documents a thirty-minute cache, which is roughly typical.

Where it came from, and why it took 28 years to become a standard

The protocol was written in 1994 by Martijn Koster, then at Nexor in Nottingham, and it came out of an ordinary problem: crawlers were new, there were very few of them, and they were knocking over servers.

Koster's position explains the design. His own search engine, ALIWEB — presented at the first World Wide Web conference at CERN in May 1994 — deliberately did not crawl at all; site owners wrote a structured file describing their own resources and registered it. His conference paper argued that robots imposed considerable network overhead, could overload servers, retrieved large volumes of irrelevant documents, and destroyed the structure of what they gathered. robots.txt is what that argument produced once it became obvious crawling would happen anyway: not a rights mechanism and not a licensing statement, but a way to keep a robot out of the parts of a site that would break under load.

It was never ratified. For nearly three decades it was a convention agreed on a mailing list, documented informally, and implemented slightly differently by everyone — which is why parsers disagreed about edge cases for so long. That changed in September 2022, when the IETF published RFC 9309, "Robots Exclusion Protocol", with Koster himself as an author alongside Gary Illyes, Henner Zeller and Lizzi Sassman of Google. The RFC invented nothing; it wrote down what the major implementations already did, so new ones could agree.

The irony is durable. The man who built a search engine on the argument that crawling was the wrong approach wrote the standard that made crawling socially acceptable, and his engine was destroyed by the crawler-based competitors it enabled.

What the protocol actually specifies

The file is a sequence of groups. Each group begins with one or more User-agent lines naming the crawlers it addresses, followed by Disallow and Allow rules giving URL path prefixes.

• A crawler obeys one group. It looks for the group naming it — matched case-insensitively on the product token, not the whole user-agent string — and falls back to the wildcard User-agent: * group only if no group names it. Rules in other groups do not apply. Mojeek documents this precisely: first MojeekBot record, else *.
• Paths are prefixes, and matching is case-sensitive. Disallow: /admin covers /admin, /administration and everything beneath; Disallow: / covers the whole host; an empty Disallow: forbids nothing.
• The most specific rule wins. Where Allow and Disallow both match, the longer matching path takes precedence — that is how a site excludes a directory while permitting one file inside it. On a tie, the least restrictive rule applies.
• Limited wildcards. * for any sequence and $ for end-of-path are near-universally supported, though both were extensions rather than part of the original convention.
• Response codes carry meaning. A 404 means no restrictions exist. A 5xx means the file is unavailable, and crawlers are expected to treat the host as disallowed rather than assume permission — which turns a misconfigured server into a temporary crawl block.
• Size is bounded. RFC 9309 requires parsers to process at least 500 kibibytes and permits them to ignore anything beyond.

Two widely used directives are not part of the standard. Crawl-delay, requesting a minimum gap between requests, is honoured by some crawlers — Naver's Yeti and You.com's YouBot — and ignored by others, including Baidu. Sitemap is also an extension, and is host-wide rather than belonging to any group.

It controls crawling, not indexing

This is the most misunderstood property of the file, and it produces a genuinely counter-intuitive outcome.

robots.txt tells a crawler not to fetch a URL. It says nothing about whether an engine may list that URL. If an engine learns about a disallowed URL from elsewhere — most commonly from links on other sites, which the exclusion rule has no reach over — it may index the URL itself, without content, and show it. The engine has obeyed the file exactly: it never fetched the page. It just did not need to fetch it to know the address exists.

Indexing is governed by a different mechanism: the noindex directive, delivered either in a meta tag inside the page or in an X-Robots-Tag HTTP header, which can also cover non-HTML files. The two controls interact badly, because noindex lives inside the response — so a crawler told not to fetch the page will never see the instruction not to index it. Blocking a URL in robots.txt and marking it noindex at the same time is self-defeating.

Two related directives sit alongside. nofollow addresses whether links are followed for discovery; nocache or noarchive addresses whether a stored copy is served. Mojeek documents support for all three alongside its robots.txt handling — a fair summary of what the well-behaved end of the field implements.

The third misunderstanding is about secrecy. robots.txt is a public file at a predictable address on every host, so a list of directories a site wishes crawlers to avoid is unavoidably a published list of directories that exist. It is a traffic-management convention, not a security boundary, and was never designed to be one.

How engines actually treat it

Compliance is not binary, and the variation between documented operators is larger than the convention's reputation suggests.

At the strict end, You.com documents full compliance including per-agent rules and Crawl-delay. Mojeek and Marginalia both publish precise handling and a blocking instruction. DuckDuckGo states that DuckAssistBot can be blocked in robots.txt and that changes take effect within 72 hours — a useful reminder that compliance has latency, because the file is cached.

Kagi introduces a subtlety: its documentation states Kagibot falls back to Googlebot directives if no Kagibot rules are present. That is a pragmatic response to a real asymmetry — many sites wrote rules for Googlebot years ago and never contemplated anyone else — and it means a site's Googlebot rules govern a crawler its author never heard of. Brave met the same asymmetry from the other side: its crawler historically did not advertise a differentiated user-agent at all, on the stated reasoning that it had to avoid discrimination from sites allowing only Google, which left publishers unable to block it cleanly until the Bravebot/1.0 token was declared.

Yandex is the most honest about partial compliance: its documentation lists roughly forty named robots and records for each whether it obeys general robots.txt directives — and some are documented as not doing so. Baidu states Baiduspider observes the file, but carries a long practitioner record of aggressive crawling and slow honouring of new exclusions. "Obeys robots.txt" is a claim with a distribution behind it, not a boolean.

The AI crawler question

Since 2023 robots.txt has been asked to carry a weight it was not built for. Koster's file expresses one idea — do not fetch this — and the argument now is not about fetching but about what fetched content may be used for. The protocol has no vocabulary for the second question.

The main operators responded by splitting their crawlers, so a single token can express a purpose. OpenAI runs GPTBot for training data, OAI-SearchBot for search visibility and ChatGPT-User for user-initiated fetches, and states that sites opted out of OAI-SearchBot will not appear in ChatGPT search answers — while blocking GPTBot does not remove them. Publishers invert this constantly. Google took the same approach with Google-Extended, a robots.txt token with no user-agent of its own governing whether content trains and grounds Gemini models, which Google states does not affect inclusion in Search.

The unresolved category is user-initiated fetching. Both ChatGPT-User and Perplexity-User are documented by their operators as agents to which robots.txt may not apply, on the reasoning that the request originates with a person rather than a crawl schedule. That is not an allegation; it is published policy. Publishers dispute the reasoning, and it sits at the centre of the current conflict. In August 2025 Cloudflare published research alleging Perplexity went further, using undeclared crawlers with generic browser user-agents to fetch pages that had explicitly blocked PerplexityBot, and removed it from its verified-bot programme; Perplexity disputed the characterisation on the same grounds.

The infrastructure layer has meanwhile stopped waiting for the protocol to catch up. Cloudflare announced in July 2025 that it would block AI crawlers by default for new domains on its network and began trialling a per-crawl payment mechanism, and in September 2025 published a content signals extension expressing machine-readable preferences about search, AI input and AI training inside robots.txt itself. Standardisation work on AI usage preferences has also been taken up at the IETF. None of it changes the fundamental property: a signal in a text file states a preference, and enforcement, where it exists, comes from a network operator refusing the connection.

What the file can and cannot settle

What it does well. It manages crawl load, keeps automated traffic out of parts of a site that generate infinite URLs or expensive responses, and gives a server administrator one universally understood place to state a preference. Thirty years on, essentially every documented search crawler reads it. As a coordination convention it has been remarkably successful — Koster's design has outlived almost every engine that first honoured it.

What it cannot do. It cannot prevent a fetch, remove a URL from an index, keep a directory secret, distinguish permitted uses of content, or bind anyone who declines to read it. It has no mechanism for consent, compensation, attribution or expiry, and no way to address a crawler nobody has named in advance.

The current moment is a mismatch of purpose. A file designed in 1994 to stop robots overloading early web servers is now the primary interface for a dispute about whether commercial models may be trained on published text. It is doing that job badly, not because the protocol is poorly designed, but because it was designed for a different question. Whether the answer turns out to be a richer signal inside robots.txt, a separate standard, network-level enforcement, licensing contracts of the kind that already govern Google's access to Reddit, or litigation, is undecided as of August 2026.

Frequently asked questions

Sources

• rfc-editor.org/rfc/rfc9309.html
• mojeek.com/bot.html
• about.marginalia-search.com/article/crawler/
• developers.openai.com/api/docs/bots
• docs.perplexity.ai/guides/bots
• yandex.com/support/webmaster/robot-workings/check-yandex-robots.html
• duckduckgo.com/duckduckgo-help-pages/results/duckassistbot
• archives.iw3c2.org/www1/PdfWWW94/aliweb.pdf