Search bots vs. training bots
AI companies run different crawlers for different jobs. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages to answer users' questions and cite sources. Training crawlers such as GPTBot and ClaudeBot collect content to train future models. You can allow one and block the other — the checker shows which is which.
Why "content in the HTML" matters
Many AI crawlers read the raw HTML and don't run JavaScript. If your text only appears after scripts load — common with some site builders and single-page apps — AI tools may see an almost empty page.
Clear beats clever
AI answers quote pages that state facts plainly: what you do, for whom, where, and for how much. Structured data, question-style headings, lists and short sentences all make your page easier to understand and cite accurately.
Questions
- Does passing this check mean ChatGPT will recommend me?
- No — it means AI tools can read and understand the page. Whether they recommend you also depends on your reputation, third-party mentions and how well you answer the question. That's what AI search optimisation works on.
- What's the reading-ease score?
- The Flesch Reading Ease score, from 0 to 100. Higher is easier: 60+ is plain English, below 30 is very hard going. It's a rough guide, not a rule.
- Is llms.txt required?
- No. It's a proposed standard that some AI tools use and others ignore. We flag it as optional — useful, cheap to add, but not a ranking factor.
More free tools