AI Crawlers

What are AI crawlers?

AI crawlers are bots that AI companies use to fetch web pages. They serve two quite different purposes, and the difference matters for your visibility:

• Training crawlers collect content to train large language models. Examples are OpenAI’s GPTBot, Anthropic’s ClaudeBot and Common Crawl’s CCBot. Google uses the “Google-Extended” token to let sites opt out of Gemini training without affecting Google Search.
• Search and retrieval crawlers fetch pages to answer questions in real time and cite them as sources – for example OAI-SearchBot for ChatGPT search, PerplexityBot, and user-triggered fetchers such as ChatGPT-User.

Website owners control access through robots.txt, by user agent. Many companies block training crawlers but allow search crawlers – blocking the latter means your pages can’t be used as AI citations. Google’s AI Overviews and AI Mode rely on the regular Googlebot, so blocking it would remove you from Google Search entirely.

Practical checks:

• Review your robots.txt and firewall or CDN bot settings: some security tools block AI crawlers by default.
• Make sure important content is in the HTML; many AI crawlers don’t execute JavaScript.
• Check your server logs to see which AI crawlers visit and which pages they fetch.

Note that robots.txt is a convention, not a technical barrier; reputable crawlers respect it, others may not.