AI crawler access checker
Checks robots.txt against a maintained catalogue of documented crawlers - search indexers, AI training crawlers, and AI assistant fetchers, each shown separately.
1 · Input
What is checked
The existing robots.txt parser evaluates access for a maintained catalogue of individually named crawler user agents, grouped into search indexing crawlers, AI training controls (like GPTBot and Google-Extended), AI assistant / user-triggered fetchers (like ChatGPT-User and Claude-User), and other documented AI crawlers - with the exact matched robots.txt group and rule shown for every one.
Precision matters here
Googlebot (Search indexing) and Google-Extended (a separate control for AI training use of Search-crawled content) are never conflated, and no two crawlers are ever collapsed into one "AI crawlers" verdict. robots.txt governs crawling only - it does not prevent every other form of use of already-published content, and a crawler being allowed here is not a guarantee it will actually visit or include the page.
.webp)
Rate this tool
Was this tool useful? Your feedback helps us improve it.