Guides
Pingory Guides: AI Readability and Crawler Policy
These guides are for people who own the server: developers, site owners and technical leads who need to know what a crawler receives, not what a vendor claims. Every page states what it verified and what it cannot prove.
Nine pages, four starting points
Start with the file most sites are missing, then the crawler list, then the blocking rules, then the measurement boundary.
| Guide | Question it answers | Minutes |
|---|---|---|
| llms.txt | What the file is and why it can be ignored | 9 |
| AI crawlers list | Which eight user agents exist | 10 |
| Block AI crawlers | Which rule breaks Googlebot | 8 |
| AI readability | What the 12 checks measure | 6 |
llms.txt: What It Is and How to Write One
The full format with a copyable example, and why a valid file can still be ignored. 9 minutes.
AI Crawlers in 2026: The Full List
Eight user-agent tokens, their owners and what each one fetches. 10 minutes.
How to Block AI Crawlers Without Breaking Google
Copyable robots.txt, Cloudflare, Vercel and nginx rules. 8 minutes.
AI Readability vs AI Visibility
The 12 checks behind the score, and the questions no tool can answer. 6 minutes.
robots.txt: The Rules Crawlers Actually Parse
RFC 9309 matching, wildcards, and why a 500 stops a crawl. 10 minutes.
JSON-LD Tester Errors: What Each Message Means
The real validator messages, mapped to a cause and a fix. 9 minutes.
XML Sitemap: What Belongs In It
The 50,000 URL limit, honest lastmod, and the retired ping endpoint. 9 minutes.
Canonical Tag: A Hint Google Can Overrule
Self-referencing tags, cross-domain targets and the noindex conflict. 8 minutes.
JSON-LD Structured Data: Fields and Nesting
@context, @type and @graph, plus placement and format choice. 8 minutes.
Start with a check
The free checker runs all 12 checks against any public domain in about ten seconds.
llms.txt Checker
Fetches llms.txt, robots.txt, sitemap.xml and the homepage, reports access for eight AI user agents, and never impersonates one.
How these guides are written
Claims here are dated and tied to a named source; where a check stops, the page says so.
- No simulated crawlers. No page here sends a forged
GPTBotuser agent and calls it a test; real verification happens by source IP. - Dates and sources. Each guide carries a verification date and names the specification behind a claim.
- Stated limits. Where a check stops, the page says so.
What none of these guides can tell you: whether an AI product will mention, cite or recommend your site. That outcome depends on the model, the question and the moment, and no file on your server records it.
Check any site free
Run the 12 checks on your own domain before changing anything.