llms.txt Checker: AI Search Readiness Audit
Audit any site's llms.txt and get an AI-readiness score with concrete fixes.
In plain language.
Checks whether a site is actually readable by AI assistants: llms.txt, robots.txt access for AI crawlers, and sitemap health.
Comes back with a 0-100 score and a plain-English list of exactly what to fix, not just a pass/fail.
Takes seconds per site — run it once before a launch, or on a schedule to catch regressions.
Most people check whether a file exists at the domain root and stop there. That misses the more common failure: a robots.txt that quietly blocks the AI crawlers regardless of what llms.txt says.
Crawler policy is resolved properly, per crawler, against robots.txt group rules including wildcards. Each result records not just whether a crawler is allowed but why — an explicit rule, a wildcard group, or the absence of a robots.txt entirely.
The score is never a black box. Every point earned and every point available comes back as a plain sentence, so the arithmetic is visible and each missing item translates directly into a task.
A site is only declared unreachable when every request fails. One missing file is a finding to act on, not an error that invalidates the audit.
The crawler set checked here is a snapshot of the significant ones at build time, not an exhaustive registry — new AI crawlers appear regularly. Where a site publishes no robots.txt at all, every crawler is reported as allowed, with the absence recorded as the reason rather than left implicit.
Scoring reflects a considered view of what matters for AI readability, published openly in each result rather than presented as an industry standard. A high score means assistants can read your site; it is not a promise that they will cite it.
Checks read only files at the domain root, and only public pages. Nothing behind a login is fetched, and existence checks read a few kilobytes rather than downloading whole files.
A sitemap declared only inside robots.txt still counts as a sitemap. Discovery follows what the standard actually permits rather than assuming a single canonical location, so a correctly configured site is not marked down for an unconventional layout.
Who it's for
- For marketing and SEO. Score your own domain alongside your main organic competitors’ and see who is already readable by AI assistants, then close your own gap before it becomes a visibility problem.
- For content and documentation teams. Audit a docs site before launch to confirm assistants can reach the pages you want cited, rather than discovering months later that robots.txt was blocking them.
- For agencies. Score a client portfolio in one run and turn the output into a prioritised remediation list, since each row carries its own reasons.
- For developers and AI agents. Run it in CI after a robots.txt change and fail the build if the score drops, so an AI-crawler block never ships unnoticed.
Pricing
$0.005 per site audited.
No subscription. Failed or empty inputs are never charged.
Fewer knobs. Better defaults.
0-100 score
A single number that's easy to track, report and improve over time.
Concrete fixes
Every failed check comes with what to change, not just that it failed.
Fast enough to run often
Cheap enough to check every site in a portfolio, regularly.
Same tool, callable as an MCP server.
llms.txt Checker runs on Apify, which means it's exposed as a remote MCP tool with no extra setup. Point any MCP-compatible agent (Claude, Cursor, and others) at the endpoint below.
View full API reference on Apify{ "url": "https://example.com" }