Guides · Crawl mechanics

robots.txt: The Rules Crawlers Actually Parse

Last verified: October 2026 · About 10 minutes of reading

Most robots.txt advice is lines to paste. This is the model underneath: which fields the standard defines, why the longest match wins, and why a 500 on one file stops a crawl.

What robots.txt is

robots.txt is a plain-text file at a host's root that states which paths a crawler may fetch; it is a request, not access control.

RFC 9309, published in September 2022, formalises a convention Martijn Koster proposed in 1994. It defines the syntax, the matching algorithm and fetch-failure behaviour — not enforcement: a crawler that ignores the file has broken a convention rather than a lock.

That distinction decides what the file is good for: it reduces crawl load on valueless paths and states intent in machine-readable form. It does not hide a page, protect a document, or remove a URL from an index.

Where the robots.txt file must live

One file per scheme, host and port, served at /robots.txt; a copy in a subdirectory or on another subdomain is never read.

The path is fixed and lowercase: a file at /folder/robots.txt is not a robots.txt file, because crawlers never look in subdirectories. Rules bind to the exact origin that served them, which produces the traps below.

Two consequences: a site that moved to HTTPS but left its old HTTP file has two independent policies, and a www-to-apex redirect does not merge files — the target's file applies after the redirect.

The four fields, and the one that is not standard

RFC 9309 defines user-agent, allow and disallow; sitemap is an outside record that crawlers may interpret, and Google does.

That vocabulary is the whole language; anything else is an extension or a comment.

Three details matter more than they look: a sitemap line must not terminate a group; a group with no rules allows everything; and Disallow: with an empty value allows every path while Disallow: / blocks every path.

# A minimal file that states a policy and nothing more
User-agent: *
Disallow: /cart/
Disallow: /search?
Allow: /search/help

Sitemap: https://example.com/sitemap.xml

Matching rules: longest match, wildcards and anchors

The most specific matching rule wins, measured in characters, so an Allow can override a broader Disallow.

Order in the file is irrelevant: RFC 9309 requires the match with the most octets, so the first rule written has no priority. Paths are case-sensitive; matching runs on the percent-encoded URL.

RFC 9309 requires support for #, $ and *. The comment character needs a warning of its own: everything after # is ignored, so a URL containing a fragment is silently truncated if pasted unescaped.

Group selection is separate from path matching: a crawler uses the group whose token matches its own, falls back to * when none does, and combines rules when several groups name it. A blank line ends a group; a missing one merges the next User-agent line into the previous group.

What happens on 404, 403 and 500

A missing file means no restrictions; a server error means a crawler may assume the whole site is disallowed.

The status code of /robots.txt is a control signal in its own right. Google documents the behaviour per status class, and the results are not symmetric: the harmless-looking failure is the one that stops a crawl.

The 5xx path is the outage that looks like a policy decision. A 503 triggers frequent retries, and without a cached copy Google assumes no restrictions. After the 30 days shown above, a reachable site counts as having no robots.txt, while one with availability problems stops being crawled. RFC 9309 is blunter: an unreachable file means assume complete disallow.

Recovery is specific: make /robots.txt return 200 with the correct body, then confirm in Search Console's robots.txt report that Google fetched it. That report has a Request a recrawl action (three-dot menu by the file) which queues an earlier fetch — it does not force one.

Size, encoding and parsing limits

The file must be UTF-8 plain text, and Google reads only the first 500 KiB of it.

RFC 9309 requires a parsing limit of at least 500 kibibytes; Google enforces exactly 500 KiB and ignores everything after it — a generated rule list past that boundary silently becomes a partial policy.

• Encoding. UTF-8 as text/plain; a BOM is ignored, and non-UTF-8 characters may be dropped in ways that break a rule.
• Line endings. CR, LF or CRLF all work; field names are case-insensitive.
• Parse errors. Invalid lines are skipped rather than failing the file, so a typo usually removes one rule, not the policy.
• HTML fallback. If an SPA answers with its HTML shell, Google extracts whatever rules it can and ignores the rest.

When a rule does not work: five causes

An ignored rule usually has one of five causes, and none is fixed by rewriting it.

1. Case sensitivity in paths. Path matching is case-sensitive: Disallow: /Admin does not cover /admin. Fields are case-insensitive; paths are not.
2. Group merging. A group is defined only by its user-agent lines, and blank lines do not end it — rules meant for one bot can leak into the group above.
3. Using robots.txt to remove a page. Disallow stops the fetch, so Googlebot never sees a noindex tag on that URL; the page can still appear without content. Blocking and de-indexing differ.
4. The edge serves a different file. A CDN, WAF or bot layer can intercept /robots.txt and return its own body — diff the fetched copy against your repository file.
5. The empty Disallow:. An empty disallow value means allow everything; a blank path from a template silently cancels the restrictions above it.

A sixth failure mode: rules blocking CSS and JavaScript. Googlebot renders pages, so a disallow on asset paths breaks rendering — Search Console reports it as blocked resources. Disallow: / is the extreme case: it stops crawling entirely.

How to test robots.txt without waiting for a crawl

Fetch the file, read the parsed rules back, and treat a 200 as proof of serving, not of obedience.

1. Fetch it as a client would. curl -sI https://example.com/robots.txt should show 200 and text/plain, with no bot challenge or login redirect in the way.
2. Read the policy, not the file. For a few real URLs, work out which rule wins by longest match. The free AI readability check reports the allow or block decision for two paths, alongside sitemap, canonical and noindex.
3. Use the engine's own parser. Search Console's robots.txt report shows the version Google fetched, its parse result and syntax warnings — the closest thing to ground truth for Googlebot.
4. Watch real requests. Server logs settle what a simulation cannot: whether the crawler arrives, and which paths it asks for afterwards.

robots.txt is not a security mechanism

The file is public and advisory; it cannot protect a document, and listing a private path advertises where it is.

RFC 9309 says the rules are not access authorization, and its security section notes that listing paths makes them discoverable — private content belongs behind authentication or a 404. A disallowed URL can still be indexed from external links, and a disallowed page's noindex can never be read, because the fetch never happens.

Which AI crawlers to allow is a policy decision, covered in the blocking guide along with the crawler identities; this page stays on the mechanics both rely on.

Frequently asked questions

Five questions cover the misunderstandings that cause the most damage: indexing, crawl-delay, server errors, subdomains and timing.

Last verified: October 2026. Parsing rules were checked against RFC 9309 and Google's robots.txt documentation on 2026-10-25; vendor behaviour outside Google is not covered by either source.