The short answer: almost certainly yes. Check your server logs for user agents like GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and ChatGPT-User — most public sites see several of them daily. They come in two kinds with very different stakes: training crawlers building future model knowledge, and retrieval fetchers pulling your pages into live answers right now. You can block any of them in robots.txt, but blocking retrieval removes you from the sources AI answers cite — for most brands, open access is the visibility-maximizing default.
Two kinds of bots, two different jobs
Training crawlers collect web content for future model training. GPTBot (OpenAI), ClaudeBot (Anthropic), and Google-Extended (Google's control for AI training use) are the big ones. Pages they collect shape what the next model versions know from memory — your contribution to tomorrow's training data. The payoff is deferred: nothing they fetch today changes today's answers.
Retrieval fetchers operate at answer time. When a user asks ChatGPT a question that warrants a web search, ChatGPT-User or OAI-SearchBot fetches pages on the spot; PerplexityBot does the same for Perplexity. These requests are your pages being read into a live answer — the mechanism of retrieval-augmented generation — and often end in a citation with your link in front of the user.
The AI crawler glossary entry keeps a fuller roster; the two-category distinction is what decisions should hinge on.
How to check what's visiting you
- Search your access logs (or CDN analytics — most CDNs now break out AI bots) for the user-agent strings above.
- Read your own
robots.txt. CMS defaults, security plugins, and old agency configs sometimes block AI agents wholesale — teams are often surprised by what they're already disallowing. - Distinguish the categories in what you find. Heavy
GPTBottraffic means you're being read for training;ChatGPT-Userhits mean real users' questions are being answered from your pages right now.
Block or allow? The actual trade-off
The case for blocking is control: some publishers object to training use of their content, and licensing-minded media companies block training crawlers as negotiating posture. If that's your situation, block selectively — robots.txt lets you disallow GPTBot while leaving OAI-SearchBot alone.
For a brand that wants to be discovered, the calculus is lopsided:
- Blocking retrieval fetchers guarantees absence. If the bot can't read your pricing page, engines answer questions about your pricing from third-party pages — accurate or not. You've ceded your own narrative to whatever else ranks.
- Blocking training crawlers erodes the long game. Future models learn about your category from a web that doesn't include your best material, while competitors' pages teach the lesson instead.
- Blocking doesn't remove you from AI answers. Engines still learn about you from everyone else's coverage; you've only silenced the one source you control.
For most brands, the sensible posture is: allow AI agents, then make what they fetch worth fetching — direct answers, current facts, structured data — per content strategy for AI search. Some sites add an llms.txt file as a curated map for these agents.
Crawling is the input; the output is what engines actually say. A free audit shows whether all that bot traffic is converting into answers that name you.