More than half of web traffic is now automated, and a fast-growing share of it is AI crawlers: training bots, answer-engine indexers, and agents fetching pages for real users. This directory documents each one — its user-agent string, its robots.txt token, how to verify it isn't spoofed, and whether blocking or monetizing it is the smarter move for a publisher.
Bulk crawlers that collect your content to train AI models. They generate no referrals and no attribution — these are the bots where blocking and licensing leverage matter most.
| Crawler | Operator | Feeds | Respects robots.txt |
|---|---|---|---|
| GPTBot | OpenAI | Training data for OpenAI's GPT models | Yes |
| ClaudeBot | Anthropic | Training data for Anthropic's Claude models | Yes |
| GoogleOther | Google research and development crawling (non-Search) | Yes | |
| Amazonbot | Amazon | Alexa question answering and Amazon AI services | Yes |
| Meta-ExternalAgent | Meta | Training data for Meta's Llama models and Meta AI | Yes |
| Bytespider | ByteDance | Training data for ByteDance's AI models (including Doubao) | No |
Crawlers that build the indexes behind AI search and answer engines. Their visits can return visibility: answers cite and link sources, so most publishers keep these allowed.
| Crawler | Operator | Feeds | Respects robots.txt |
|---|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT search results and link citations | Yes |
| Claude-SearchBot | Anthropic | Claude's web search index and citations | Yes |
| PerplexityBot | Perplexity | Perplexity's answer-engine search index | Reported issues |
| Googlebot | Google Search — and, via Search, AI Overviews and AI Mode | Yes | |
| Bingbot | Microsoft | Bing Search — and, via Bing's index, Microsoft Copilot | Yes |
| Applebot | Apple | Siri and Spotlight suggestions; Applebot-Extended governs Apple Intelligence training | Yes |
| DuckAssistBot | DuckDuckGo | DuckAssist AI answers in DuckDuckGo search | Yes |
On-demand fetchers that retrieve a single page when a real person asks an AI assistant about it. Each hit is a human reading your content through an agent — invisible to your analytics.
| Crawler | Operator | Feeds | Respects robots.txt |
|---|---|---|---|
| ChatGPT-User | OpenAI | Live page fetches when ChatGPT users browse or ask about a URL | Yes |
| Claude-User | Anthropic | Live page fetches when Claude users ask about a URL | Yes |
| Perplexity-User | Perplexity | Live page fetches when Perplexity users ask about a URL | Not for user requests |
Archive crawlers whose public datasets downstream AI companies train on. One robots.txt line here can have wider reach than blocking any single AI lab.
| Crawler | Operator | Feeds | Respects robots.txt |
|---|---|---|---|
| CCBot | Common Crawl (nonprofit) | The Common Crawl public web archive — a foundation of many AI training datasets | Yes |
Not crawlers at all: robots.txt-only control tokens that govern how already-crawled content may be used. They never appear in your logs.
| Crawler | Operator | Feeds | Respects robots.txt |
|---|---|---|---|
| Google-Extended | Gemini model training and Gemini app grounding | Yes |
Each crawler page gives you the exact user-agent string, the robots.txt token with copy-paste snippets, the official IP ranges or verification method (user-agent strings are trivially spoofed), and an honest block-or-monetize assessment. A useful rule of thumb across the whole directory: allow the search indexers and user-triggered fetchers that cite you, gate the training crawlers that don't. And remember that robots.txt is a request, not enforcement — for the bots that ignore it, control has to happen at the network edge. Details were last reviewed August 2026; verify against operator documentation before hardcoding firewall rules.
Oasy detects and fingerprints 50+ AI crawlers with per-URL analytics, blocks the ones you exclude at the edge, and turns the rest into revenue — licensed RAG access and sponsored placement inside AI answers, settled weekly. Analytics scripts can't see this traffic; your server logs can, and so can we.
Join the waitlistStay informed on life in Europe as an expat. The Local delivers daily news, guides and essential info across 9 European countries. Sign up for the free newsletter at thelocal.com/free-newsletter