What an AI crawler means
An AI crawler is a bot that fetches web pages to feed AI systems rather than a traditional search index. If your site has any public footprint, these bots are almost certainly in your server logs already. They come in two functionally different kinds, and the difference matters more than the label:
- Training crawlers collect pages to build the corpora future models learn from. What they gather today shapes what next year's models "know by heart" about your brand — your training data footprint.
- Retrieval fetchers pull pages at answer time, when an engine grounds a live response in current sources via retrieval-augmented generation. What they fetch can appear — cited — in an answer minutes later.
The crawlers worth knowing
| User agent | Operator | Primary purpose |
|---|---|---|
GPTBot | OpenAI | Training data collection |
ChatGPT-User | OpenAI | Live fetching for ChatGPT answers |
OAI-SearchBot | OpenAI | ChatGPT search indexing |
ClaudeBot | Anthropic | Training data collection |
PerplexityBot | Perplexity | Search indexing and retrieval |
Google-Extended | Controls Gemini training use of content |
Each publishes its user-agent string and honors robots.txt, so access is yours to configure. Google-Extended is a control token rather than a separate bot — regular Googlebot does the crawling; the token governs whether the content trains Gemini.
Allowing vs blocking: the real trade
Blocking is a one-line robots.txt rule per bot:
User-agent: GPTBot
Disallow: /
But the two crawler types carry different stakes. Blocking training crawlers is a content-licensing stance with a slow visibility cost: your site stops contributing to what future models know about you (though third-party coverage still does). Blocking retrieval fetchers has an immediate cost: your pages can no longer be fetched into live answers, which removes your domain from the sources engines cite and lets competitors' content answer your category's questions. Publishers with paid content may accept that trade; for most brands trying to be recommended, allowing retrieval fetchers is the visibility-maximizing default. See do AI chatbots crawl my website for how to audit your logs and decide.
Managing AI crawlers in practice
The working posture for a brand pursuing AI visibility: explicitly allow the bots you want (absence of a rule means allowed, but being deliberate survives future edits), keep key pages fast and server-rendered so fetchers see full content without executing JavaScript, and optionally add an llms.txt file to point AI systems at your canonical pages. Then verify the outcome rather than assuming it — whether engines are actually citing your domain shows up in citation-rate tracking, which is the ground truth on whether your crawl policy is translating into presence in answers.