llms.txt Checker

Free, no account required. Last reviewed: October 2026.

This checker inspects a public domain and tells you whether an AI crawler can actually read it. It fetches the files and markup a crawler receives — /llms.txt, /robots.txt, /sitemap.xml and the homepage HTML exactly as it is served without running JavaScript — and reports what is missing. We call the result AI Readability: a measurement of whether the door is open and the sign is legible. It is not a ranking, a citation forecast, or a promise that any AI product will quote you.

What is llms.txt?

llms.txt is a plain-Markdown file at /llms.txt that gives AI systems a curated map of a site: what the site is, which pages matter, and where to find them. Jeremy Howard of Answer.AI proposed it in September 2024 as a community convention. It is not a W3C recommendation, an IETF standard, or a vendor requirement, and nothing enforces it.

The format is intentionally small: one H1 with the site or product name, an optional one-line summary as a blockquote, then H2 sections that group ordinary Markdown links. Because it is plain text, it is cheap to serve and easy to diff between deploys.

What it is not: a crawl blocker, a substitute for robots.txt, or a guarantee of inclusion in any model. Those distinctions matter, and the rest of this page keeps them explicit.

Check your llms.txt

Enter a bare domain. One free check takes about ten seconds and looks at the files above plus the crawler rules that apply to eight named AI crawlers.

What this checker actually verifies

Twelve checks run against the public site. Eight are core: if any core check fails, the overall status becomes down. The other four are enhancements — they never change the status, only the score. Weights add up to 100.

Weights and scope of each check.
Check	Weight	What it verifies
robots.txt	8	The file is reachable as plain text, parses as RFC 9309 groups and rules, and is not an HTML fallback page.
AI crawler matrix	22	Eight named AI crawler tokens are allowed on both the homepage and a deeper path. Training group: GPTBot, ClaudeBot, Google-Extended, CCBot. Retrieval group: OAI-SearchBot, ChatGPT-User, Claude-User, PerplexityBot. Google-Extended and CCBot are control tokens the operator acts on rather than standalone fetchers, so they are evaluated as policy, not as traffic.
llms.txt	6	The file exists, is not empty, and has at least one H1 and one Markdown link. Presence alone is not treated as a pass.
JSON-LD	12	Every structured-data block parses, carries `@context`, and declares at least one `@type` (including inside `@graph`).
First-screen text	12	How much text is present in the raw HTML before JavaScript runs. A JavaScript-only shell is reported as such.
sitemap.xml	6	A valid `urlset` or `sitemapindex` with at least one entry, or one declared in robots.txt.
noindex	18	Meta robots and `X-Robots-Tag` are parsed token by token. `noindex` is a failure; narrower restrictions are warnings.
canonical	4	The canonical URL resolves to this page, on the same host and protocol.
Meta title & description	5	Title and description exist and fall inside sane length ranges. Enhancement only.
Open Graph	3	`og:title`, `og:description` and `og:image` are present. Enhancement only.
H1	2	Exactly one non-empty H1. Enhancement only.
Image alt text	2	Images that are not explicitly decorative carry an alt attribute. Enhancement only; skipped when a page has no images.

The checker never impersonates an AI crawler. Cloudflare and similar networks verify bots by source IP, so a forged GPTBot user agent would be blocked and produce a false negative. Everything here is derived from the publicly served files and HTML.

llms.txt vs robots.txt vs sitemap.xml

These three files are often confused. Only one of them can actually stop a crawler.

Different jobs, different levels of enforcement.
Aspect	llms.txt	robots.txt	sitemap.xml
Purpose	A curated Markdown map of the pages worth reading.	Rules for which paths a crawler may fetch.	An inventory of URLs with optional last-modified dates.
Status	Community proposal, September 2024. Not a standard.	RFC 9309. The convention every major crawler documents.	Sitemaps.org protocol. Widely supported.
Location	`/llms.txt`	`/robots.txt`	`/sitemap.xml`
Can it block crawling?	No.	Yes — this is the only layer crawlers are documented to obey.	No.
Who acts on it	No major AI vendor has publicly committed to reading it.	Search and AI crawlers, by documented convention.	Search engines and some AI crawlers, for discovery.
Failure mode	Ignored silently.	Misconfigured rules block the wrong agent.	Stale URLs waste crawl budget.

How to write a valid llms.txt

Keep it short and curated. The checker requires an H1 and at least one Markdown link; everything else is judgement.

Serve it at https://your-domain/llms.txt as text/plain; charset=utf-8.
Start with one H1 naming the site or product.
Add a one-line blockquote summary so a reader knows what the site is before following links.
Group links under H2 headings. Use absolute URLs with a short description after each link.
Link the pages that matter — docs, pricing, API reference, changelog — not every URL in your sitemap.
Re-check the file after a deploy. A CMS or SPA fallback route can quietly replace it with HTML.

Example

# Example Site

> Example Site helps developers ship faster. Docs, API reference and changelog.

## Documentation
- [Getting started](https://example.com/docs/start): Install and send your first request
- [API reference](https://example.com/docs/api): Every endpoint and response shape

## Product
- [Pricing](https://example.com/pricing): Plans, limits and what is included
- [Changelog](https://example.com/changelog): Release notes, newest first

## Optional
- [Status page](https://example.com/status): Current availability

The five mistakes we see most

Serving the app shell instead of the file. A catch-all route answers /llms.txt with 200 text/html and the SPA's index page. The checker flags this as an HTML response or invalid format.
No H1 or no links. A paragraph of prose is not a map. Without a heading and at least one link there is nothing for a reader to follow.
Dumping the whole sitemap. Ten thousand URLs is not curation. A useful file points at the pages that represent the site.
Treating llms.txt as access control. It cannot block anything. If you need a crawler to stay out, the rule belongs in robots.txt.
Writing it once. Pages move, products get renamed, and the file silently rots. Re-check it after structural changes.

Do AI crawlers actually read llms.txt in 2026?

No major AI vendor has publicly committed to reading llms.txt as of 2026. OpenAI, Anthropic, Google and Perplexity document robots.txt and their own user-agent behaviour; none of them documents llms.txt as a requirement or a supported control. So the honest answer is narrower than the marketing: no vendor documentation promises to read it, and we do not publish numbers we cannot verify.

The mechanism crawlers are documented to obey remains robots.txt under RFC 9309. That is why this page talks about AI Readability: it measures whether your content is readable and your policy is legible, not whether an answer engine chose to cite you. A valid llms.txt is a cheap, honest signal. It is not a lever.

One more distinction the checker keeps: training crawlers and retrieval crawlers are not the same audience. Training crawlers are GPTBot, ClaudeBot, Google-Extended and CCBot. Retrieval crawlers are OAI-SearchBot, ChatGPT-User, Claude-User and PerplexityBot. Blocking the training group while allowing the retrieval group is a mainstream, defensible policy in 2026 — it limits future model training without removing you from answers today. This checker reports that combination as a policy note, not a failure.

Monitor llms.txt over time

A single check is a snapshot. The failures that hurt are the quiet ones: a deploy replaces robots.txt, a plugin starts answering /llms.txt with HTML, or a retrieval crawler gets caught by a new firewall rule.

Pingory's AI monitor type re-runs this check on your schedule and alerts you when the result changes, alongside uptime, TLS expiry and keyword checks. AI monitors are included in the Pro plan at $6/month; Starter ($4/month) covers uptime, keyword, DNS and TLS checks.

Watch llms.txt, robots.txt and AI crawler access automatically.

Create an account, add an AI monitor on the Pro plan, and get an email the moment readability changes.

Start free

Frequently asked questions

Is llms.txt an official standard?

No. Jeremy Howard of Answer.AI proposed it in September 2024 as a community convention. There is no W3C recommendation, IETF RFC, or vendor mandate behind it, and no AI company is obliged to read it.

Where should I put llms.txt?

At the root of the domain: https://example.com/llms.txt. This checker only inspects that exact root path, so a copy under a subpath or on a subdomain is not evaluated here.

What Content-Type should llms.txt use?

text/plain is the safe choice, and text/markdown is also reasonable. Serve it as UTF-8. If your server returns text/html, you are probably serving the single-page-app fallback instead of the file.

Does llms.txt replace robots.txt?

No. robots.txt controls crawling and is the mechanism crawlers are documented to obey under RFC 9309. llms.txt is a content map with no enforcement. Keep robots.txt as your access-control layer and treat llms.txt as a summary.

Does OpenAI's ChatGPT read llms.txt?

OpenAI has not publicly committed to reading llms.txt. Its documented crawl controls are robots.txt and its own user-agent rules. Treat llms.txt as a hint, not a guarantee.

Do AI crawlers respect robots.txt?

The major crawlers document that they do, and RFC 9309 is the convention they follow. Enforcement is voluntary in practice, so a crawler can ignore it, but robots.txt is still the only mechanism with broad, documented support.

Why does the checker say "Unknown"?

Some sites sit behind a firewall or bot challenge that blocks our checker. When that happens we cannot inspect the page, so the score is unknown, not zero. Check your firewall logs or allow the Pingory-AIReadability user agent.

Is this llms.txt checker free?

Yes. A single check is free and needs no account. To keep it available for everyone we limit each network to 10 checks per 10 minutes.

How often should I check my llms.txt?

After any deploy that changes robots.txt, the sitemap, or your key pages. A monthly spot check catches slow drift, such as a stale link or a CMS plugin rewriting the file.

Can I monitor llms.txt automatically?

Yes. Pingory's AI monitor type re-runs this check on a schedule and alerts you when the result changes. AI monitors are included in the Pro plan ($6/month), not in Starter.

Method: each check issues up to five HTTP GET requests to the public site (homepage, robots.txt, llms.txt, sitemap.xml, plus any sitemap declared in robots.txt) with our own identifying user agent, follows up to five redirects with an SSRF guard on every hop, and parses only what is returned without executing JavaScript. Limits: this is a single-region check; a firewall that blocks our crawler produces an unknown result, not a failure.