AI crawler checker
See which AI and search crawlers a site's robots.txt lets in. A store that blocks the search crawlers behind ChatGPT, Claude or Perplexity cannot be read or linked in their answers.
Free, no sign-up. Reads the site's public robots.txt once. Works on any site, not just yours.
Three kinds of AI crawler
Each AI company runs more than one crawler, and they do different jobs. Blocking one does not block the others, so a robots.txt written to keep content out of training can leave search open, or close it by accident.
- Search crawlers put a site into answers and link to it. OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity, Googlebot for Google Search and Applebot for Siri and Spotlight. These are the ones a store wants in.
- Training crawlers collect pages that may train models. GPTBot and ClaudeBot crawl for it. Google-Extended and Applebot-Extended are not crawlers at all, only a say in whether pages already crawled may be used. Google says Google-Extended does not affect Google Search.
- User fetchers open a page when someone asks about it. ChatGPT-User, Claude-User and Perplexity-User. OpenAI and Perplexity say theirs may not follow robots.txt, because a person asked for the page.
Each crawler follows the group in robots.txt that names it. With no group of its own it follows the User-agent: * rules, and Apple says Applebot follows Googlebot's rules when robots.txt names Googlebot but not Applebot. The checker shows which group decided each answer.
Changing it on Shopify
Shopify writes robots.txt for you. To change it, add a robots.txt.liquid template to the theme, as Shopify describes in editing robots.txt. Shopify calls this an unsupported customisation and warns that a mistake can cost all search traffic, so change only the groups you mean to. On other platforms, robots.txt is a plain file at the root of the site.
What this can and cannot tell you
What it reads
- The site's robots.txt, every group and rule in it
- For each of twelve crawlers, whether the homepage is open to it
- On Shopify, product, collection and blog pages as well
- Which group decided each answer: its own, the * group, or Googlebot's for Applebot
What it cannot see
Whether a firewall, CDN or bot setting turns a crawler away before robots.txt is ever read, which is common with Cloudflare. It cannot see whether any AI answer actually cites the site, and it reads no page content, tags or tracking. User fetchers such as ChatGPT-User may not follow robots.txt at all, by their owners' own account.
Questions people ask
How do I check if ChatGPT can access my website?
Enter the address above and look at OAI-SearchBot, which is the crawler OpenAI uses to show sites in ChatGPT search. If it is blocked, ChatGPT search cannot read the site. GPTBot is a separate crawler for training, and blocking it does not remove a site from ChatGPT search, according to OpenAI.
Should a store block GPTBot?
That is a choice about training, not visibility. GPTBot collects pages that may train OpenAI models. Blocking it keeps your pages out of that, and OpenAI says it does not stop OAI-SearchBot showing you in ChatGPT search. The same split holds for ClaudeBot and Claude-SearchBot.
Does blocking Google-Extended remove my site from Google?
No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It only controls whether Gemini may train on, or be grounded in, pages Google has crawled.
My robots.txt allows everything, so why do AI tools still not show my site?
robots.txt only says what crawlers may read. A firewall or CDN can still block them, which robots.txt cannot show, and being readable does not mean an answer engine chooses to cite you. On Cloudflare, check the bot settings for AI crawlers.
How do I edit robots.txt on Shopify?
Add a robots.txt.liquid template to your theme under Online Store, then Edit code. Shopify keeps its default rules in that template, so add or change only the groups you need. Shopify warns that a mistake can block search engines entirely.