Do you let AI answer engines crawl your site?

Most sites treat AI crawlers as one group. They block all of them, or allow all of them, in a single robots.txt rule. Both choices cost you something, because the crawlers do different jobs.

Some collect content to train models. Others fetch a page to answer a user's question and cite the source. Blocking the first group is a licensing decision. Blocking the second group removes you from AI answers entirely, which is rarely what anyone intended.

Know which crawler does what

Two jobs, two very different consequences.

Retrieval crawlers fetch pages to answer questions and build search indexes. Blocking one costs you citations and referral traffic. OpenAI and Anthropic both document what each of their crawlers does.

Training crawlers collect content to train models. Blocking one has no effect on whether you are cited today.

Do not block the crawlers you want traffic from

This is the most common mistake, and it is usually accidental. A site adds a blanket AI block, then wonders why it never appears in ChatGPT.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: PerplexityBot
Disallow: /

❌ Figure: Bad example - Bad example - One blanket rule that also blocks every retrieval crawler

# Retrieval: allowed, so we can be cited
User-agent: OAI-SearchBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Training: not allowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /

✅ Figure: Good example - Good example - Retrieval allowed, training denied

If you want both, allow both. The point is to decide deliberately rather than block by reflex.

Google-Extended does not affect AI Overviews

This myth costs people real traffic, so it is worth stating plainly.

Google-Extended controls whether your content is used for Gemini training and grounding. It does not control whether you appear in AI Overviews or AI Mode. Those are features of Google Search, and they follow ordinary Googlebot crawl rules and snippet settings. See Google's overview of Google crawlers.

Note the reverse trap too. A nosnippet or max-snippet:0 directive does remove you from AI Overviews, because those answers reuse the same snippet text.

State your position with Content Signals

robots.txt can say who may fetch a page. It cannot say what they may do with it afterwards. The Content Signals Policy adds that missing half, with three independent signals.

• search - may appear in search results
• ai-input - may be used to ground an AI answer
• ai-train - may be used to train a model

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

✅ Figure: Good example - Good example - A clear position on all three uses

Treat Content Signals as a declaration of intent rather than an enforcement mechanism. It states your position in a machine-readable way; it does not stop anyone.

Check what you are actually publishing

Read your own robots.txt and confirm it matches the decision you think you made. Watch for these traps:

• A blanket Disallow: / in a User-agent: * group applies to every crawler that has no group of its own
• The most specific user-agent group wins, and it replaces the * group rather than adding to it
• Within a group, the longest matching path wins, and Allow beats Disallow on an equal-length match
• A staging Disallow: / copied to production blocks everything, including Googlebot

The precedence rules are defined in RFC 9309, the Robots Exclusion Protocol.