Dattva Blog · October 13, 2026

robots.txt for AI Crawlers: GPTBot, ClaudeBot, and PerplexityBot Explained

By Madhu — Founder, Dattva AI | October 13, 2026

GPTBot, ClaudeBot, and PerplexityBot are the three primary AI crawlers a robots.txt file should explicitly allow, each requiring its own named User-agent rule, since a generic wildcard rule written for one crawler does not reliably apply to the others.

Why Each AI Crawler Needs Its Own Rule

Treating all AI crawlers as one category in a robots.txt file misses that each one is a separately named user agent. A rule written for one does not automatically apply to another unless a wildcard rule is used on purpose. In practice, a site owner reads about ChatGPT visibility, explicitly allows GPTBot, and assumes the job is done. ClaudeBot and PerplexityBot stay exactly as blocked or unblocked as they were before.

I see this half-fix often enough that I now check all three crawlers by name before looking at anything else on a site. If you want the simplest version of this check first, start with how to check whether ChatGPT is blocked from crawling your website, then come back here for the full configuration.

What Each of the Three Major Crawlers Actually Does

GPTBot is OpenAI's crawler, used to gather content that may inform ChatGPT's responses and training. ClaudeBot serves a similar function for Anthropic's Claude. PerplexityBot supports Perplexity's retrieval-based answer generation, which leans on crawling content closer to real time than crawlers that mainly feed periodic training updates.

OAI-SearchBot is a related but distinct OpenAI crawler used for search-style retrieval rather than general training data collection. Google-Extended is a token that controls whether content can be used for Google's AI features, separate from the standard Googlebot used for regular search indexing. Each of these needs its own explicit decision in a robots.txt file. Assuming one covers the others creates partial, inconsistent access, which shows up as uneven AI visibility across platforms.

What the Data Shows About Crawler Access

An analysis of AI citation patterns found that 73% of sites carry technical barriers that block AI crawler access (OtterlyAI, February 2026). A robots.txt rule is only one of those barriers, but it is the easiest one to inspect and the easiest to get wrong.

Crawl volume makes the stakes higher. ChatGPT sends 3.6 times more crawl requests to websites than Googlebot does (Alli AI via Search Engine Journal, 2026), so the traffic your robots.txt is deciding on is no longer a side channel.

When I review a site, the wildcard rule is the usual culprit. Someone wrote a broad Disallow years ago to deal with scrapers, and it quietly catches every AI crawler that is not separately allowed above it. This is the pattern I flag most often in audits, and I treat it as a working observation, not a measured rate.

How to Write a Correct robots.txt Configuration

List each AI crawler as its own User-agent group with an explicit Allow, placed above any broader wildcard rules. Most parsers apply the most specific matching group, so the order and the specificity both matter.

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: *
Disallow: /admin/
Disallow: /cart/

Allow the whole site unless specific sections genuinely need to stay private. Keep the wildcard group narrow, limited to paths that no crawler should touch, so it cannot catch AI crawlers by accident.

After editing, test the live file at yourdomain.com/robots.txt and confirm each named crawler resolves to an Allow, using a robots.txt validator. Then confirm the change is not undone one layer up. A firewall or CDN can still block these crawlers before robots.txt is ever read, which I cover in how default security settings accidentally block AI crawlers.

What Other Crawlers Are Worth Including

Amazonbot is worth including for brands with retail or product-related content, since Amazon's own AI features draw on it. New named crawlers keep appearing as more platforms add AI search, so a robots.txt file is not a one-time setup. Review it every quarter and after any hosting, CDN, or CMS change, since those are the moments rules get overwritten without anyone deciding to.

For the wider technical review that sits around this file, a developer's checklist for AI crawler accessibility walks through robots.txt, security layers, and rendering together. Dattva's ongoing AI visibility monitoring also re-checks crawler access on a schedule, so a rule that gets changed later does not stay unnoticed for months.

Conclusion

Configuring robots.txt correctly for AI crawlers means treating GPTBot, ClaudeBot, PerplexityBot, and related bots as individually named visitors, each with its own explicit Allow rule. It is a small file with an outsized effect on whether a brand's content can be considered for AI citation at all. Check it first, before any content work, because no amount of good writing helps a page the crawler was never allowed to read.

Frequently Asked Questions