AI Crawlers: OAI-SearchBot, GPTBot, ClaudeBot and PerplexityBot

What Are AI Crawlers?

AI crawlers are automated bots operated by AI companies that visit websites to collect information. This information is used to build search indexes, generate answers, or train AI models.

The Main AI Crawlers

GPTBot (OpenAI)

User agent: GPTBot

Purpose: Collects data for OpenAI's models and products

Documentation: https://platform.openai.com/docs/gptbot

OAI-SearchBot (OpenAI)

User agent: OAI-SearchBot

Purpose: Powers ChatGPT's search feature

Note: Separate from GPTBot; specifically for real-time search

ChatGPT-User (OpenAI)

User agent: ChatGPT-User

Purpose: Used when ChatGPT browses the web in real time during a conversation

ClaudeBot (Anthropic)

User agent: ClaudeBot

Purpose: Collects data for Anthropic's Claude models

PerplexityBot (Perplexity)

User agent: PerplexityBot

Purpose: Indexes content for Perplexity's AI search engine

Google-Extended (Google)

User agent: Google-Extended

Purpose: Used for Gemini and other Google AI products (separate from Googlebot)

Bingbot (Microsoft)

User agent: Bingbot

Purpose: Powers Bing search and Microsoft Copilot

How to Configure Access

In your robots.txt file, you can allow or block specific crawlers:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

To block a crawler:

User-agent: GPTBot
Disallow: /

Recommendation

If you want AI engines to find and cite your business, allow their crawlers. Block only private or sensitive pages (admin, dashboard, API endpoints).

Sources

OpenAI GPTBot documentation

Anthropic ClaudeBot documentation

Perplexity developer documentation

Google AI crawler documentation