GLOSSARY
AI crawler
(GPTBot · AI bot)
- AI crawler
- An AI crawler is a bot an AI company uses to fetch web pages — for model training (GPTBot, ClaudeBot), for a search index (OAI-SearchBot, PerplexityBot), or to retrieve a page live during a user's question. Blocking the search and user-fetch bots removes you from AI answers regardless of content quality.
Each AI vendor now runs a fleet of crawlers split by job, and the split is the point: blocking training bots is a legitimate policy choice with no visibility cost, while blocking search-index and live-fetch bots silently removes you from citations. OpenAI runs GPTBot (training), OAI-SearchBot (search index) and ChatGPT-User (live fetch); Anthropic runs ClaudeBot and Claude-SearchBot; Perplexity runs PerplexityBot and Perplexity-User. Google AI Overviews use ordinary Googlebot — there is no separate bot to court — while Google-Extended only controls Gemini training.
The scale is significant: AI crawlers plus AI-search bots account for roughly 27% of verified bot traffic (Digital Applied, from Cloudflare data). A typical visibility-safe robots.txt policy looks like:
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Training bots — your policy decision, no citation impact:
# User-agent: GPTBot
# Disallow: /The most common finding in our audits is accidental blocking: a CDN bot rule or security plugin denying every non-Google crawler, which zeroes out AI visibility before content is even considered. Crawler access is the first check in the AI visibility audit because every other fix is irrelevant until it passes.