// AI CRAWL CHECKER
Checks whether your site allows AI search and citation bots (the ones that get you cited in ChatGPT, Claude, and Perplexity answers), and separately, whether it allows AI training bots. These are different questions with different answers.
If the page is blank or shows a 404, your site doesn't have one yet. That's fine, it just means the default (everything allowed) applies. Use the generator further down to create one.
Auto-check may fail since most servers block cross-site requests from a browser. If it does, use Open Robots.txt above, then copy and paste the result below.
Note: robots.txt is a request, not a lock. Most major bots honor it, but not all do. A separate file, llms.txt, is a growing convention for guiding AI agents to your best content, but it doesn't control access the way robots.txt does.
The AI Crawl Checker reads your robots.txt and shows whether AI search and citation bots are allowed to visit your site, and whether AI training bots are blocked. It checks eight crawlers in two groups:
Each bot gets one of three results. ALLOWED means your robots.txt doesn't block it. BLOCKED means a rule disallows your whole site for that bot. NOT SET means no rule names it and no general rule applies, so the default (everything allowed) is in effect.
Allowing a bot to cite your site is a different choice from letting it train on your content, and robots.txt lets you decide each one on its own. Many sites choose to stay visible in AI answers while opting out of training. The checker scores the two groups separately for that reason, and the generator above starts with search bots allowed and training bots off, so you opt in to training only if you want to.
If you want to go a step further and point AI tools to the pages you most want them to read, see the llms.txt generator. For other fixes that affect how crawlers read your site, the schema markup generator and the broken link checker are in the lab too.
That's your call, and it depends on how you feel about your content being used to train models. Blocking training bots in robots.txt asks those crawlers not to collect your pages for training. It doesn't stop AI search bots from citing you as long as those stay allowed, which is why this checker treats the two groups separately.
Then the default applies and every bot is allowed to crawl everything. That's why a missing file shows as NOT SET. If you want to block training bots or set specific rules, use the generator on this page to create a starter robots.txt.
No. robots.txt is a request, not a lock. Major crawlers from the big AI companies say they respect it, but a bot that ignores the rules can't be stopped by this file alone. If you need real access control, that has to happen on the server.
robots.txt controls which bots may crawl your site. llms.txt is a newer convention for pointing AI tools to your most useful content. It doesn't control access, so the two work together rather than replacing each other.