Guides · llms.txt

llms.txt: What It Is, How to Write One, and Whether AI Actually Reads It

Last verified: October 2026 · About 9 minutes of reading

Most guides explain the syntax and stop there. This one also covers the part that decides whether the file does anything: what a crawler is actually obliged to do with it, which is nothing. You will get a complete, copyable example, the exact serving requirements, and a way to test your own file without guessing.

What llms.txt is

llms.txt is a plain-text Markdown file at /llms.txt that names the pages you want an AI system to read first.

Jeremy Howard of Answer.AI proposed it in September 2024 as a community convention. It is not a W3C recommendation, not an IETF RFC and not a requirement from any AI vendor. Nothing enforces it, and no company has to fetch it.

The file is small on purpose: a title, one line about the site, then a handful of links grouped by purpose. Being plain text, it costs almost nothing to serve and diffs cleanly between deploys.

What problem it was meant to solve

The proposal targets a real gap: HTML navigation is noisy, and a crawler has to guess which pages carry the substance.

A typical marketing site buries four useful pages under cookie banners, mega-menus, tag archives and paginated listings. A sitemap does not help much either: it lists every URL with equal weight and says nothing about which ones are worth a fetch. llms.txt states that judgement explicitly — here are the ten pages that describe this product, in reading order.

It is a reading list, not an access-control list. The distinction matters later, when people expect the file to keep crawlers out.

The format, field by field

The format is one H1, an optional blockquote summary, then H2 sections holding Markdown links with optional notes.

There is no schema and no validator with authority. The proposal describes a shape, and every element below is a convention rather than a rule. In practice, these are the parts that matter:

• H1 — the site or product name, once. A file without a heading gives a reader nothing to anchor on.
• Blockquote summary — one line, directly after the H1. It tells the reader what the site is before any link is followed.
• H2 sections — groups such as Documentation, Product, Optional. The last one is a convention for links that can be skipped when context is short.
• Link list items — - [Title](absolute URL): one-line note. Absolute URLs only; relative paths depend on a base the reader may not have.

A complete example you can adapt as-is:

# Example Site

> Example Site is a hosted API for invoice parsing. Docs, pricing and changelog below.

## Documentation
- [Getting started](https://example.com/docs/start): Install the SDK and send a first request
- [API reference](https://example.com/docs/api): Every endpoint, parameter and error code
- [Webhooks](https://example.com/docs/webhooks): Retries, signatures and delivery order

## Product
- [Pricing](https://example.com/pricing): Plans, quotas and overage rules
- [Changelog](https://example.com/changelog): Release notes, newest first

## Optional
- [Status page](https://example.com/status): Current availability and incident history
- [Security](https://example.com/security): Data retention and subprocessors

Two details are worth copying. Every link carries a short note, because a note is the only place the file can explain why the page matters. And the sections are ordered by what a newcomer needs first.

Where to put it and how to serve it

Serve it at the domain root as /llms.txt with HTTP 200 and the header text/plain; charset=utf-8.

The path is literal. https://example.com/llms.txt is the file; https://example.com/docs/llms.txt is a different URL that a root-only reader will never request. If you run several subdomains, each host needs its own copy, because there is no cross-host include mechanism.

The most common serving failure is a single-page app or catch-all route answering /llms.txt with 200 text/html and the app shell. The status code looks fine and the content is wrong. Two ways to prevent it:

# nginx — serve the real file and never fall through to the SPA
location = /llms.txt {
    default_type "text/plain; charset=utf-8";
    add_header X-Content-Type-Options nosniff;
    try_files /llms.txt =404;
}

# Cloudflare Pages / Netlify — public/_headers
/llms.txt
  Content-Type: text/plain; charset=utf-8

Then confirm what the network actually returns, not what your editor shows:

curl -sI https://example.com/llms.txt | head -n 5
# HTTP/2 200
# content-type: text/plain; charset=utf-8

curl -s https://example.com/llms.txt | head -n 3
# # Example Site

If the second command prints <!DOCTYPE html>, the file is not being served. Fix that before touching the content.

How it differs from robots.txt

robots.txt controls access and is obeyed by convention; llms.txt only suggests what is worth reading.

These two files are routinely confused, and the confusion produces bad advice. One of them can keep a crawler out. The other cannot.

So the rule is simple: policy goes in robots.txt, inventory goes in sitemap.xml, judgement goes in llms.txt. A file that tries to do the first job will fail, because readers treat it as advisory text.

Adoption in 2026

As of October 2026, no major AI vendor publicly commits to reading llms.txt, so adoption is intent rather than a guarantee.

The publishing side is active: CMS plugins generate the file, documentation themes ship one, and directories list sites that have it. Real requests for /llms.txt do appear in server logs, sometimes from AI-adjacent agents, sometimes from SEO crawlers collecting the file as a signal.

The consuming side is quieter. OpenAI, Anthropic, Google and Perplexity document robots.txt and their own user-agent rules; none of them documents llms.txt as a supported input or a requirement. A request in a log proves that something fetched the file. It does not prove that the file shaped an answer, and Pingory does not publish percentages it cannot verify.

Why a correctly formatted file can still be ignored

A valid file can be ignored because nothing obliges a crawler to fetch it, parse it, or prefer the links inside.

Five mechanisms explain most of the gap between "we published it" and "it did something":

1. No enforcement layer. robots.txt is followed because crawler operators chose to follow it and said so publicly. llms.txt has no equivalent commitment, so a crawler that skips it is not violating anything.
2. Crawlers already have sources. Sitemaps, internal links and rendered HTML give a retrieval system enough to work with. A second map is a convenience, not a necessity.
3. Retrieval is query-driven. When a person asks a question, the system fetches pages that match the question. Your curated home page may never be the best match for that specific query.
4. Serving errors. The file returns HTML, 404, a redirect chain, or plain text behind a bot challenge. Anything that is not a 200 with text is a silent failure.
5. Policy beats preference. If robots.txt disallows the crawler, the reading list is irrelevant. Access control always wins over a suggestion.

None of this makes publishing pointless: a small one-time cost, an unmeasurable upside, and a file you can verify in ten seconds.

How to verify yours

Verify the file the way a crawler would: exact path, status 200, plain text, one H1 and at least one link.

Run these four checks after every deploy that touches routing:

• The URL is https://your-domain/llms.txt, with no trailing slash and no redirect.
• The response is 200 and the content type starts with text/plain.
• The body starts with a single # heading and contains at least one Markdown link.
• Every link resolves. A 404 inside the file is worse than no entry, because it wastes the one fetch you may get.

If you would rather not do that by hand, the free llms.txt checker fetches the same files a crawler would and reports what is missing. It runs 12 checks, of which llms.txt is one worth 6 points; the crawler-access matrix is worth 22 and usually explains a low score. That checker is a single check with no account. Recurring monitoring is a different product, covered in AI readability.

Frequently asked questions

These five questions cover the parts readers ask about most: status, location, citation, policy and size.

Last verified: October 2026. Facts on this page were checked against the original llms.txt proposal, RFC 9309 and vendor crawler documentation on 2026-10-24.