Googlebot — Google's ai search indexer

Googlebot renders JavaScript, crawls with sophisticated scheduling based on your site's freshness and authority, and remains the most important crawler for human traffic on the web.

The strategic change is downstream: material crawled by Googlebot can now be synthesized into AI Overviews that answer the query above the organic links. Controls like nosnippet and data-nosnippet limit AI Overview usage, but they also degrade your normal search snippets — Google offers no clean 'search yes, AI answers no' switch.

Googlebot identifies itself with the following user-agent string:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Never trust the user-agent string alone. Scrapers routinely impersonate well-known crawlers to inherit their access. Google publishes Googlebot's IP ranges as JSON and supports reverse-DNS verification (hosts under googlebot.com / google.com). Spoofed Googlebot is among the most common fake-bot traffic on the web.

Published IP ranges: https://developers.google.com/search/apis/ipranges/googlebot.json

Note that most AI crawlers and fetchers, Googlebot included, do not execute JavaScript — so this traffic is invisible to GA4 and every script-based analytics tool. Server logs, CDN analytics, or a dedicated bot-analytics layer are the only places you will see it. For the full picture of measuring AI-driven visits, see our guide on how to track AI traffic.

To refuse Googlebot access to your entire site, add this to your robots.txt:

User-agent: Googlebot
Disallow: /

To restrict it from specific sections only (for example, premium content) while leaving the rest open:

User-agent: Googlebot
Disallow: /premium/
Disallow: /members/

The case for blocking: Almost no publisher can afford to block Googlebot outright. The realistic lever is snippet controls on your most valuable content, accepting the cost to normal search presentation.

The case for allowing or monetizing: You already monetize Googlebot — it's called SEO. The new problem is that AI Overviews capture a growing share of clicks that used to reach you; the response is measuring AI-era visibility and building presence inside answers, not just under them.