Block the trainer, not the source of your citations.

Why the difference between these bots matters

Most blocking advice dumps everything with AI in the name into one list with Disallow under it. That is exactly the wrong move. Blocking a training crawler protects your content from being reused inside a model. Blocking a search crawler or user-initiated fetcher means you never appear as a source in an answer again. Same directive, opposite outcome.

1. Training

OpenAI describes GPTBot as the crawler that collects content which may train its foundation models. Anthropic says the same of ClaudeBot: it collects web content that could contribute to training. CCBot belongs to Common Crawl, which is not an AI company, but the public archive it builds has been one of the largest training sources for years. Google-Extended and Applebot-Extended belong here too, with one difference: they are control tokens rather than crawlers. Block this group and you lose no visibility.

2. Grounding and retrieval

This is the group that earns you citations. OpenAI states that OAI-SearchBot surfaces websites in search results in ChatGPT and is not used for training. Anthropic says Claude-SearchBot navigates the web to improve search result quality. Perplexity describes PerplexityBot as designed to surface and link websites in Perplexity results, explicitly not for foundation models. Block this group and you disappear from the answers.

3. User-initiated fetches

ChatGPT-User, Claude-User and Perplexity-User fetch a page because a human just asked a question or pasted a link. No index, no training: it is a visitor with an intermediary. Blocking these is the same as turning a customer away because somebody else drove them to your door.

Who is who, and what does blocking cost?

The table below summarises the official documentation from OpenAI, Anthropic, Perplexity, Google and Apple. Read each row as: does this bot earn me citations, and does it produce training data?

Which robots.txt suits you?

There are three realistic postures. Pick one deliberately, and remember that robots.txt is a request rather than a lock: well-behaved crawlers honour it, bad actors do not.

A. Allow everything (the default for almost every small business)

If you want customers from AI search and your content has no paid archive value, this is the right call. A missing or empty robots.txt already means everything is allowed; write it out anyway, with your sitemap alongside.

User-agent: *

Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

B. Allow retrieval and citation, block training

The defensible middle ground for publishers and knowledge-heavy businesses: be named in answers without donating the training material. Every user agent needs its own block.

# Training: no

User-agent: GPTBot

Disallow: /

User-agent: ClaudeBot

Disallow: /

User-agent: CCBot

Disallow: /

User-agent: Google-Extended

Disallow: /

User-agent: Applebot-Extended

Disallow: /

# Citing and fetching: yes

User-agent: OAI-SearchBot

Allow: /

User-agent: ChatGPT-User

Allow: /

User-agent: Claude-SearchBot

Allow: /

User-agent: Claude-User

Allow: /

User-agent: PerplexityBot

Allow: /

User-agent: Perplexity-User

Allow: /

User-agent: Bingbot

Allow: /

User-agent: *

Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

C. Block everything

Only sensible when your content is the product and every AI reference costs you revenue. The price: you vanish as a source from ChatGPT, Claude and Perplexity, including when an existing customer explicitly asks about you.

User-agent: GPTBot

User-agent: OAI-SearchBot

User-agent: ChatGPT-User

User-agent: ClaudeBot

User-agent: Claude-SearchBot

User-agent: Claude-User

User-agent: PerplexityBot

User-agent: Perplexity-User

User-agent: CCBot

User-agent: Google-Extended

User-agent: Applebot-Extended

Disallow: /

The block you never see

This is the most important warning on this page. Blocking is rarely a deliberate robots.txt decision; it happens by accident, one layer down, in your WAF or bot management.

In July 2025 Cloudflare switched on AI crawler blocking by default for new domains. It also ships a managed robots.txt with an Enforce robots.txt rules option that deploys a WAF rule, and that rule blocks at the network layer regardless of what your own robots.txt says. Bot Fight Mode additionally challenges traffic that does not look like a browser, which is precisely what a crawler looks like.

The cruel part is that nothing tells you. Search Console only reports on Googlebot, so a 403 served to GPTBot or PerplexityBot never shows up there. Your robots.txt says Allow, your analytics say nothing, and meanwhile no AI fetcher gets through. Verify at the edge, not in the file.

So should I publish an llms.txt?

You may, it takes five minutes, and there is no measured return today. In May 2026 Ahrefs studied 137,000 domains and found that 97% of llms.txt files received zero requests. Of the requests that did arrive, AI retrieval bots accounted for 1.1%.

The vendors say the same thing. John Mueller stated in June 2025 that no AI system currently uses llms.txt, and Google’s own documentation is explicit: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities)." Near-zero cost to ship, near-zero measured return. Anyone promising results from llms.txt is selling something nobody can demonstrate.

Do not forget Bing

ChatGPT search leaned on the Bing index for a long time; OpenAI additionally crawls with its own OAI-SearchBot. Submitting your sitemap to Bing Webmaster Tools takes ten minutes and is one of the few levers you genuinely control here. So do not block Bingbot for good measure while tidying up AI bots.

How FWRD handles this

Every FWRD site is open to search and retrieval crawlers by default and submits its sitemap automatically. There is no bot management sitting in front of it that accidentally stops AI fetchers, and the pages are static HTML: what a crawler retrieves is what a visitor sees.