Free tool · No signup

AI Crawler Checker: can AI bots access your site?

Check your live robots.txt against 19 AI and search crawlers, from GPTBot and OAI-SearchBot to ClaudeBot, PerplexityBot and Google-Extended. See which bots are allowed, partly restricted or blocked, and what that means for your AI search visibility.

Quick answer

To be cited in AI search, allow the search and user-request crawlers: OAI-SearchBot and ChatGPT-User (ChatGPT), Claude-SearchBot and Claude-User (Claude), PerplexityBot and Perplexity-User (Perplexity), plus Googlebot and Bingbot. Training crawlers such as GPTBot, ClaudeBot and CCBot, and the Google-Extended token, can be blocked separately without removing you from AI search results.

Crawler list last reviewed: October 2026

Bot types

The four kinds of AI crawler

Not every AI bot does the same job. Knowing which is which lets you allow AI search while making a separate decision about training.

Search engine crawlers

Googlebot, Bingbot and Applebot. They power Google AI Overviews, Microsoft Copilot and Apple features. Blocking them removes you from search.

AI search indexers

OAI-SearchBot, Claude-SearchBot and PerplexityBot build the indexes AI answer engines cite from.

User-request fetchers

ChatGPT-User, Claude-User and Perplexity-User fetch a page when a user asks about it in real time.

Training crawlers and tokens

GPTBot, ClaudeBot, CCBot, Bytespider, plus the Google-Extended and Applebot-Extended control tokens, govern model training.

Reference

AI crawler and user agent list

These are the robots.txt tokens the checker tests. Search and user-request bots decide whether you can be cited in AI answers; training bots and control tokens decide whether your content may be used to train models.

robots.txt token	Operator	Type	Used for
`Googlebot`	Google	Search engine	Google Search, AI Overviews, AI Mode. Crawls for Google Search, which includes AI Overviews and AI Mode. Blocking it removes your pages from Google.
`Google-Extended`	Google	Control token	Gemini apps, Vertex AI. Not a separate crawler. A robots.txt token that decides whether content Googlebot fetches may be used to train and ground Gemini models. It does not affect Google Search or AI Overviews.
`Bingbot`	Microsoft	Search engine	Bing Search, Microsoft Copilot. Crawls for Bing Search, whose index also grounds Microsoft Copilot answers.
`OAI-SearchBot`	OpenAI	AI search index	ChatGPT search. Indexes pages so they can be shown and linked in ChatGPT search results.
`ChatGPT-User`	OpenAI	User request	ChatGPT, custom GPTs. Fetches a page when a ChatGPT user or a GPT action asks for it. It is not used for automatic crawling.
`GPTBot`	OpenAI	Model training	OpenAI model training. Crawls content that may be used to train OpenAI foundation models. Blocking it does not remove you from ChatGPT search.
`Claude-SearchBot`	Anthropic	AI search index	Claude search. Indexes content to improve the quality of search results in Claude.
`Claude-User`	Anthropic	User request	Claude. Fetches pages when a Claude user asks a question that needs them.
`ClaudeBot`	Anthropic	Model training	Anthropic model training. Crawls content that may be used to train Anthropic models.
`PerplexityBot`	Perplexity	AI search index	Perplexity answers. Indexes websites so they can be surfaced and linked in Perplexity answers. Perplexity states it is not used to train foundation models.
`Perplexity-User`	Perplexity	User request	Perplexity. Fetches a page when a Perplexity user asks a question that needs it.
`Applebot`	Apple	Search engine	Siri, Spotlight, Safari. Crawls for Apple search features in Siri, Spotlight and Safari.
`Applebot-Extended`	Apple	Control token	Apple generative models. Not a separate crawler. A token that decides whether content Applebot fetches may be used to train Apple generative models. It does not stop Applebot crawling.
`DuckAssistBot`	DuckDuckGo	AI search index	DuckDuckGo AI answers. Fetches pages to answer questions in DuckDuckGo AI-assisted answers.
`Amazonbot`	Amazon	AI search index	Alexa and Amazon services. Crawls to improve Amazon products and services, including answers given by Alexa.
`MistralAI-User`	Mistral AI	User request	Le Chat. Fetches pages when a Le Chat user asks for them.
`Meta-ExternalAgent`	Meta	Model training	Meta AI and model training. Crawls content for AI training and AI features in Meta products.
`CCBot`	Common Crawl	Model training	Common Crawl open dataset. Builds the open Common Crawl web archive, which many language models are trained on.
`Bytespider`	ByteDance	Model training	ByteDance AI models. Crawls content for ByteDance AI products and model training.

Operators add and rename tokens. Confirm against each operator's own crawler documentation before changing your robots.txt.

How to allow AI search but block AI training

Most sites want to be found and cited in AI answers. Some also want to keep their content out of model training. Because search and training use different tokens, you can do both. This robots.txt allows AI search and user requests, and opts out of the main training crawlers:

# AI search and user requests: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# Model training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Bytespider
Disallow: /

# Everyone else
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Two details catch many sites out. First, a bot that has its own User-agent group ignores the * group completely, so a Disallow you wrote for everyone does not apply to it. Second, the longest matching rule wins, and when an Allow and a Disallow are the same length, Allow wins. The checker applies both rules exactly as RFC 9309 describes.

Search bots vs training bots

Platform	Needed to be cited	Training only
ChatGPT	OAI-SearchBot, ChatGPT-User	GPTBot
Claude	Claude-SearchBot, Claude-User	ClaudeBot
Perplexity	PerplexityBot, Perplexity-User	None published
Google AI Overviews and AI Mode	Googlebot	Google-Extended (control token)
Microsoft Copilot	Bingbot	None published
Apple	Applebot	Applebot-Extended (control token)

Common mistakes the checker catches

Blocking everything with User-agent: * and Disallow: /, often left over from a staging site. Every AI search bot inherits it.
Blocking OAI-SearchBot or PerplexityBot by accident while trying to block training crawlers.
Blocking Google-Extended expecting to leave AI Overviews. AI Overviews are part of Google Search and use Googlebot.
A robots.txt that returns HTML, such as a theme's 404 page, which contains no valid rules.
A robots.txt that returns a 5xx error, which tells compliant crawlers to stay away from the whole site.

How to confirm AI bots are actually visiting

robots.txt tells bots what they may do; your server logs show what they actually do. Search your access logs for the user agent tokens above, such as OAI-SearchBot, PerplexityBot or Claude-SearchBot. Regular requests mean the bot is reading your pages. If a bot is allowed but never appears, check that your CDN or firewall is not challenging or blocking it.

Crawler access is only the first step

Allowing AI bots means they can read your pages. Whether they cite you depends on what they find: clear answers, explicit entities, structured data and independent sources that confirm your claims. The free GEO Score Checker measures those signals, the AI visibility checker tests whether AI answers cite you today, and the llms.txt checker validates the optional file some AI tools read.

AI crawler questions

What is an AI crawler checker?

An AI crawler checker reads your live robots.txt and reports whether the bots used by AI search engines and AI companies are allowed to fetch your pages. This one tests 19 user agents, including OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot, Googlebot, Bingbot and Google-Extended, and applies the rules exactly as RFC 9309 describes.

How do I check if ChatGPT can crawl my website?

Enter your domain in the checker above. It reports whether OAI-SearchBot (ChatGPT search), ChatGPT-User (pages fetched on a user's request) and GPTBot (model training) are allowed. To appear in ChatGPT search results, OAI-SearchBot must be allowed.

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, and pages users ask about are fetched by ChatGPT-User. You can block GPTBot to opt out of training while still allowing OAI-SearchBot so your pages can be cited in ChatGPT search.

What is Google-Extended and does blocking it hurt my Google rankings?

Google-Extended is a robots.txt control token, not a separate crawler. It decides whether content Googlebot fetches may be used to train and ground Gemini models. Blocking it does not affect Google Search rankings or whether you appear in AI Overviews, which are part of Google Search and depend on Googlebot.

Which AI bots should I allow?

If you want visibility in AI answers, allow the search and user-request bots: Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. Training crawlers such as GPTBot, ClaudeBot, CCBot and Google-Extended are a separate business decision about whether your content may be used to train models.

Should I block AI training bots?

It depends on your goals. Blocking training crawlers keeps your content out of future model training but does not remove you from AI search, as long as the search and user-request bots stay allowed. Publishers often block training; most businesses that want brand awareness allow it.

What happens if my robots.txt returns an error?

Under RFC 9309, a missing robots.txt (4xx) means crawlers may access everything. A server error (5xx) means crawlers should assume the whole site is disallowed until the file is reachable again. A robots.txt that returns an HTML page contains no valid rules. The checker reports each of these cases.

Do all AI bots obey robots.txt?

robots.txt is a voluntary standard. The major operators listed here publish their user agent tokens and state that their automated crawlers respect robots.txt, but some user-initiated fetchers follow different rules, and unknown scrapers may ignore it entirely. For hard blocking, use your firewall or CDN bot controls.

Is robots.txt the same as llms.txt?

No. robots.txt controls which URLs crawlers may access. llms.txt is an optional Markdown file that points language models to your key pages; it grants or blocks nothing. Validate it with the llms.txt checker.