AI Crawler Management: The 2026 robots.txt Policy That Works

GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Applebot-Extended, Google-Extended, CCBot, Meta-ExternalAgent, Amazonbot, cohere-ai — the AI crawler list keeps growing, and on a growing share of sites the combined bot traffic from these user-agents now exceeds Googlebot. Most robots.txt files do not mention a single one of them. That gap is the post.

The reason it matters: an AI bot is not a search engine. It can scrape for training, fetch live for retrieval, or both, and the user-agent string alone does not tell you which. Allowing the wrong bot burns crawl budget, leaks unpublished drafts, and feeds your competitors’ models. Blocking the wrong one makes you invisible inside ChatGPT, Perplexity, and Gemini — surfaces an increasing share of discovery now happens on. A robots.txt policy is the only file that decides both, and almost nobody audits it.

This is the framework we use, the policy we ship, and the evidence we look at to decide which AI bots deserve which treatment on a given site. The honest summary up front: there is no universal right answer, but there is a working decision tree, and the robots.txt you wrote in 2022 is no longer it.

Key takeaways

• AI crawler traffic now rivals search-engine traffic on many sites. A typical content-heavy blog sees 30–60% of crawler hits from non-search bots, with GPTBot, ClaudeBot, PerplexityBot, and CCBot leading.
• One user-agent often masks two products. OpenAI’s GPTBot crawls for training, while OAI-SearchBot and ChatGPT-User fetch live for retrieval. Same vendor, opposite intent, different opt-out decisions.
• Google-Extended is not Googlebot. Blocking it does not affect Google Search rankings. It controls whether your content trains Gemini, and nothing else.
• The default policy is wrong on both sides. A blanket User-agent: * block cuts you out of every AI assistant overnight; a blanket allow feeds training pipelines you may not want to feed.
• The honest framework is a 5-tier ladder: training-only bots blocked, retrieval-only bots allowed, search-affiliated bots allowed with citation rules, scraper-class bots blocked at the edge, and unidentified bots rate-limited.
• Trust the logs, not the press releases. Most “AI search” bots announce a user-agent but ship their real traffic through a different one. Verification is the same reverse-DNS workflow as for Googlebot.
• A robots.txt is not a contract. It is a polite request. Hard control lives at your edge layer or WAF — the post covers where the file stops working and what to layer on top.

The new bot landscape

In 2022 a robots.txt named three or four bots worth caring about: Googlebot, Bingbot, Slurp, Baiduspider. The rest were noise. In 2026 the file needs to handle a bot census that reads like a tech-conference badge list, and the rules around what each one does are not written down in one place. Two vendors split their bots across multiple user-agents on purpose, and that split is the first thing to understand.

OpenAI ships at least three:

• GPTBot — crawls for model training.
• OAI-SearchBot — fetches live for ChatGPT search results.
• ChatGPT-User — fires when a ChatGPT user clicks through to your site, same role as a regular visitor.

Same vendor, three different intents. Blocking GPTBot but allowing OAI-SearchBot keeps you out of training while staying in ChatGPT search. Blocking all three cuts you out of both.

Anthropic follows the same pattern: ClaudeBot for training, Claude-Web for retrieval, plus the legacy anthropic-ai label on older systems. Perplexity ships PerplexityBot for retrieval (it does not train on your pages today) and Perplexity-User for citation clicks. Google splits off Google-Extended specifically so publishers can opt out of Gemini training without affecting Google Search. Apple ships Applebot-Extended for the same reason. Meta uses meta-externalagent for its AI surfaces; Amazon uses Amazonbot.

The pattern is consistent: the search-indexing crawler stays under one brand, and a separate -Extended or training-specific user-agent handles the AI-training use case. Knowing which is which is the entire policy.

What each AI bot actually does with your content

Marketing copy from these vendors is consistent in tone and inconsistent in mechanism. The practical summary, drawn from each vendor’s published documentation and the log data we see on client sites:

• GPTBot — crawls for OpenAI training. Honours robots.txt opt-out. Independent of ChatGPT search retrieval.
• OAI-SearchBot — fetches live results for ChatGPT search. Independent of training. Same robots.txt treatment as a search crawler.
• ChatGPT-User — fires only when a ChatGPT user clicks a citation. Treated as a real user in analytics; do not block.
• ClaudeBot — crawls for Anthropic training. Honours robots.txt.
• Claude-Web — live retrieval for Claude search answers. Citations may include your URL.
• PerplexityBot — live retrieval and citation; does not train on your pages as of 2026. Citation attribution is partial — sometimes a link, sometimes not.
• Google-Extended — controls Gemini training. Disabling it does not affect Google Search indexing or rankings. This is the single most misunderstood user-agent in the file.
• Applebot-Extended — controls Apple Intelligence training. Disabling it does not affect Apple Search.
• CCBot — Common Crawl. Used by most major model trainers as raw training data. Honoured by everyone downstream.
• meta-externalagent — Meta AI training and retrieval.
• Amazonbot — Amazon’s AI surfaces ( Rufus, Alexa). Mixed role.
• cohere-ai, cohere-training-data-crawler — Cohere training and retrieval.
• DuckAssistBot — DuckDuckGo’s AI assistant retrieval.

The user-agent list is the only signal you have for intent. A site that wants to be cited in AI answers but not used as training fuel blocks GPTBot/ClaudeBot/meta-externalagent/CCBot and allows OAI-SearchBot/Claude-Web/PerplexityBot/Applebot-Extended (the retrieval-only ones, which mostly don’t train). The opposite policy — let the models learn from you but do not cite you — is achievable in principle and pointless in practice.

The 5-tier policy framework

The decision tree that holds up across site types is a 5-tier ladder. Each tier is a robots.txt stance plus an edge-layer stance, and the answer for any given bot is determined by answering four questions in order.

1. Does this bot train a model on my content? If yes, decide whether you want to be training data. Most content sites say no to closed-source training and yes to retrieval — that is tier 2.
2. Does this bot retrieve live content for AI answers and cite me? If yes, you almost always want it in. Citation is the AI-era equivalent of ranking.
3. Is this bot a search engine in disguise? Treat it as one — honour indexability signals, allow rich snippets, but verify it follows the same rules as Googlebot before trusting it.
4. Is this bot unidentified or scraping-class? Block at the edge. robots.txt is a request, not a guard.

The five tiers in order:

• Tier 1 — Search-engine bots (Googlebot, Bingbot, DuckDuckBot, Applebot, Slurp). Full allow with rich-snippet and preview signals. Same treatment you already have.
• Tier 2 — AI retrieval bots (OAI-SearchBot, ChatGPT-User, Claude-Web, PerplexityBot, Perplexity-User, DuckAssistBot). Full allow. They drive citation traffic and discovery, which is the AI-era ranking.
• Tier 3 — AI training bots (GPTBot, ClaudeBot, meta-externalagent, cohere-training-data-crawler, Google-Extended, Applebot-Extended). Allow if your business model benefits from being training data; block otherwise. There is no right answer — it is a commercial choice.
• Tier 4 — Aggregator crawlers that feed multiple trainers (CCBot). Treat as Tier 3 by default; CCBot’s data ends up in dozens of models, so blocking it covers most of the long tail.
• Tier 5 — Unknown, scraper-class, or unverifiable bots. Block at the edge with rate limits and DNS verification. robots.txt alone will not stop them.

The mental model that ties the tiers together: every bot on the list is either a search engine, a citation source, a training pipeline, or noise. Decide which one is which, then route the policy accordingly.

The robots.txt we ship

Here is the policy SEOKit24 runs in production, abridged from the live file. The full thing lists every named bot individually so that any future bot not yet on the list falls into the catch-all block — the inverse of the typical “allow all by default” stance.

# AI crawlers — explicitly allowed for training + retrieval (cite-with-link)
# These bots power ChatGPT search, Perplexity, Claude, Gemini, Bing Chat.
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: CCBot
Allow: /

# Default rule for any other crawler
User-agent: *
Allow: /

Three things to notice:

• No blanket User-agent: * at the top. The default rule sits at the bottom, after the named bots, so the explicit Allow entries take precedence by specificity (a rule most engines honour — but not all, which is why tier 5 still needs the edge layer).
• No Disallow other than internal paths. A blanket block is one mistake away from making the site invisible to a newly announced bot that has not been added to the named list yet.
• GPTBot and OAI-SearchBot are separate. Treating them as one is the single most common OpenAI mistake in robots.txt files.

The reasoning for the allow stance: this site is a free toolkit. Citation in an AI answer drives direct traffic and earns attribution; training-fuel use is acceptable cost. A publisher with paywalled content or proprietary data would flip tier 3 and 4 to a block. A publisher whose customers come through AI citations would lean the other way. The policy is the same shape, the direction is yours.

Common robots.txt mistakes that quietly cut you out of AI search

The mistakes we see in audits, in order of how often they show up:

Treating all AI bots as one user-agent

The most common error. A team adds User-agent: GPTBot and assumes they have covered OpenAI — they have not, because OAI-SearchBot and ChatGPT-User are separate products with separate policies. Same for Anthropic’s three-user-agent split and Apple’s training-vs-search split.

Adding Disallow: / under the wrong user-agent

A User-agent: * with Disallow: / does what it says on the tin: it blocks every crawler including future AI bots that have not been named yet. Teams often add this thinking they are blocking scrapers; they are also blocking their own AI-search citation.

Assuming Google-Extended affects Google Search

It does not. Google-Extended is a separate opt-out for Gemini training. Disabling it changes nothing about whether your pages rank in Google Search. If a vendor or pitch deck tells you blocking Google-Extended improves your Google rankings, that vendor is wrong.

Confusing polite request with enforcement

A bot with a research-paper agenda and no commercial incentive to behave can ignore robots.txt. CCBot honours it. Hundreds of smaller AI scrapers do not. robots.txt is a request, not a guard. Anything sensitive needs an edge-layer rule, not a robots directive.

Using Noindex where they meant Disallow

A Noindex is a page-level signal telling Google not to index that URL. A Disallow is a robots-level signal telling compliant bots not to fetch it. The two are not interchangeable, and using Noindex inside a robots.txt does nothing — robots.txt does not carry that directive.

Robots.txt is the file that decides which bots see your site. The AI era just made the bot list longer than anyone expected, and the 2022 policy no longer covers it.

How to verify what AI bots actually hit your site

The press release says one thing. The logs say another. Verification is the same workflow as for Googlebot, applied to the new list.

1. Pull 14 days of access logs. The non-engineer workflow is the one we already use elsewhere — a spreadsheet, a pivot table, a per-user-agent sort.
2. Filter by AI bot user-agents. The list above is the working set. Add new ones as they appear; a bot claiming to be AI-Search-Bot/1.0 with no documentation goes into a separate “unverified” bucket.
3. Verify IPs the same way as Googlebot. Forward and reverse DNS. Real GPTBot resolves to *.openai.com; real ClaudeBot resolves to *.anthropic.com. Anything else is spoofed.
4. Group by URL. The same top-1% crawl-bucket pattern shows up — AI bots disproportionately hit your most recent and most-linked content, with a long tail of low-value URLs they discovered through outbound links from elsewhere.
5. Compare against your policy. If you blocked GPTBot and the logs show Mozilla/5.0 (compatible; GPTBot/1.0) at volume, the bot is spoofing its user-agent or routing around robots.txt — and your edge layer needs to step in.

The output of the verification step is rarely what teams expect. On most sites we audit, the volume split between training bots, retrieval bots, and spoofed scrapers is roughly 40/40/20, with the spoofed 20% doing most of the damage. The fix is at the edge, not in robots.txt.

When robots.txt is not enough

Three situations require more than a robots.txt:

• High-volume spoofed scrapers. They ignore robots.txt. The fix is at your edge layer or WAF — rate-limit by ASN, block by reverse-DNS failure, challenge by JA3 fingerprint.
• Authenticated content. Anything behind a login is invisible to compliant bots by default, but a leaked URL or a shared session token reaches AI bots through legitimate retrieval paths.robots.txt cannot fix leaked access.
• Content licensing. If you sell syndication or your data carries contractual use restrictions, robots.txt is a starting position, not a contract. The legal layer is your terms of service and your CDN access controls, not the file at /robots.txt.

For everything else, robots.txt is the right tool, and the framework above is the right policy shape.

How to ship a robots.txt change without breaking SEO

The decision to block an AI bot is a business decision; the rollout is a technical one. The order matters.

1. Stage the change. Deploy the new robots.txt to a staging origin first, or serve it via a X-Robots-Tag rewrite at the edge so you can flip back in a minute.
2. Watch the indexing layer. Search Console Performance for 7–14 days; sudden impression drops mean the new policy caught a search-engine bot you did not name.
3. Watch the AI citation layer. Track referrals from chat.openai.com, perplexity.ai, claude.ai, gemini.google.com, copilot.microsoft.com in your analytics. A citation drop is the AI-search equivalent of a ranking drop.
4. Re-verify after every CMS template change. robots.txt rot is real; a CMS migration that rewrites /robots.txt is how sites accidentally get blocked.

The full audit cadence lives in our broader technical SEO framework — the technical SEO guide walks the crawl-to-index pipeline end to end, and the log file analysis playbook shows you how to read what the bots actually did versus what they claim.

The decision most teams get wrong

The most expensive decision in this file is not which bot to block. It is treating robots.txt as a one-time deliverable. New AI bots ship every quarter — OpenAI’s searchbot family grew three-fold in 18 months — and the file you wrote last year is missing entries you have never heard of. The audit cadence for robots.txt should be quarterly, not annual, and every time you do it the work is the same: pull logs, verify the bot list, decide each tier, ship the change, watch the next 14 days.

The second-most expensive decision is treating the policy as a moral question. It is a commercial one. “Should AI be allowed to train on the open web” is a debate worth having. “Should this site be cited when a user asks ChatGPT about SEO audits” is the question robots.txt actually answers, and the answer for most content sites is yes.

Frequently asked questions

Does blocking GPTBot affect ChatGPT search rankings?

No. GPTBot handles training, not retrieval. OAI-SearchBot handles ChatGPT search. If you want to be cited in ChatGPT search results, you need to allow OAI-SearchBot — blocking GPTBot does not affect it either way.

Does blocking Google-Extended hurt Google Search rankings?

No. Google-Extended is a separate opt-out covering Gemini training. Google Search rankings are controlled by Googlebot, not Google-Extended. The two are intentionally separate so publishers can opt out of training without affecting search visibility.

Should I block all AI crawlers by default?

Probably not. A blanket User-agent: * block cuts you out of every AI assistant overnight. The economic cost of being uncited in AI answers is already larger than the cost of being included in training pipelines, and the gap widens as ChatGPT, Perplexity, and Gemini absorb more discovery traffic. The 5-tier framework above is the working middle ground.

How do I know if an AI bot is following my robots.txt?

User-agent alone is not proof. Verify with reverse DNS the same way you verify Googlebot. Real GPTBot resolves to *.openai.com; real ClaudeBot resolves to *.anthropic.com. Anything else is spoofed and your edge layer is the only fix. The reverse-DNS workflow is the same one our log file analysis post walks through for Googlebot.

Is robots.txt legally enforceable?

Not on its own. It is a polite request, not a contract. Hard enforcement lives in your terms of service, your CDN access controls, and your edge-layer rules. For sensitive content, robots.txt is a baseline, not the answer.

How often should I audit my robots.txt?

Quarterly. New AI bots ship every few months, your own CMS templates change, and the AI-citation landscape moves faster than any annual audit can keep up with. Pair the audit with the cadence from our Search Console hidden reports walkthrough — the same reports that catch indexation drift will catch robots.txt drift too.

The verdict

The 2026 robots.txt is a longer file than the 2022 one, and the decision is harder. There is no universal policy; there is a 5-tier framework, a verification workflow, and a quarterly audit cadence. The teams getting this right treat the file as a living document, not a launch deliverable, and they route the answer to “should this bot see my content” through the same logs-not-press-releases evidence they already use for Googlebot.

The bots will keep multiplying. The framework will hold. Verify the user-agents, route by tier, watch the logs, and ship the change quarterly. That is the policy that actually works.

Want to see how your current robots.txt stacks up? Run it through the Sitemap & Robots Validator and audit every named bot against the tier list above — the file ships with a per-bot check, so the gaps show up in seconds.