ClaudeBot: What It Is, User Agent & Should You Block It?

What is ClaudeBot?

ClaudeBot crawls broadly to build training corpora and has drawn publisher attention for high request volumes during active crawl periods. It does not execute JavaScript.

Anthropic states that ClaudeBot respects robots.txt directives. Older Anthropic crawl activity used the tokens 'anthropic-ai' and 'claude-web'; some publishers keep those legacy tokens disallowed as well for completeness.

How to identify ClaudeBot

ClaudeBot identifies itself with the following user-agent string:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

Never trust the user-agent string alone. Scrapers routinely impersonate well-known crawlers to inherit their access. Anthropic documents its crawlers and their behavior in its help center; genuine ClaudeBot traffic identifies itself with the ClaudeBot user-agent and a contact address. Verify unusual volumes against reverse DNS and published guidance before whitelisting.

Note that most AI crawlers and fetchers, ClaudeBot included, do not execute JavaScript — so this traffic is invisible to GA4 and every script-based analytics tool. Server logs, CDN analytics, or a dedicated bot-analytics layer are the only places you will see it. For the full picture of measuring AI-driven visits, see our guide on how to track AI traffic.

Controlling ClaudeBot with robots.txt

To refuse ClaudeBot access to your entire site, add this to your robots.txt:

User-agent: ClaudeBot
Disallow: /

To restrict it from specific sections only (for example, premium content) while leaving the rest open:

User-agent: ClaudeBot
Disallow: /premium/
Disallow: /members/

Worth knowing

Anthropic's three bots are controlled independently: ClaudeBot (training), Claude-SearchBot (search index), Claude-User (on-demand fetches). Blocking one does not block the others.

Should you block or monetize ClaudeBot?

The case for blocking: The case mirrors GPTBot: training access produces no referrals, no attribution, and no revenue, while consuming your content's value. Blocking preserves licensing leverage.

The case for allowing or monetizing: Anthropic has signed content licensing deals with publishers, which means the market rate for training access is demonstrably above zero. Gating ClaudeBot and offering structured, licensed access is the coherent strategy for content businesses.

Bottom line

Block by default unless you have a licensing arrangement. Keep Claude-SearchBot and Claude-User allowed if you want Claude to cite and read your pages for real users.

Where Oasy fits

See exactly what ClaudeBot does on your site — then decide what that access is worth.

Oasy detects and fingerprints 50+ AI crawlers with per-URL analytics, blocks the ones you exclude at the edge, and turns the rest into revenue — licensed RAG access and sponsored placement inside AI answers, settled weekly. Analytics scripts can't see this traffic; your server logs can, and so can we.

Join the waitlist

Frequently asked questions

Does blocking ClaudeBot remove my site from Claude's answers?+

It removes your content from future training data, but Claude can still cite you via Claude-SearchBot's index and fetch pages on user request via Claude-User — each has its own robots.txt token.

Does ClaudeBot respect robots.txt?+

Yes — Anthropic documents that ClaudeBot honors robots.txt, and publisher logs bear this out. It also identifies itself with a contact email in the user-agent string.

What are the anthropic-ai and claude-web user agents?+

Legacy tokens from Anthropic's earlier crawling. Current collection runs under ClaudeBot, but many publishers disallow the old tokens too — there is no downside to doing so.

Stay informed on life in Europe as an expat. The Local delivers daily news, guides and essential info across 9 European countries. Sign up for the free newsletter at thelocal.com/free-newsletter