AI Crawlers and robots.txt: What to Allow and Block
Which AI crawlers to allow in robots.txt, from OAI-SearchBot to ClaudeBot and PerplexityBot, and how to stay visible in AI answers while opting out of training.
If you want your SaaS to show up in ChatGPT, Claude or Perplexity answers, allow each company's search crawler in robots.txt. If you also want to keep your pages out of model training, block the training crawler separately. The two jobs use different bots, so you can do both.
Most founders hear "block AI bots" and slam one door. That often removes you from AI search too. Here is how to split the decision properly.
Why do AI companies run more than one crawler?
Each company separates three jobs: collecting content for training, indexing pages for search answers, and fetching a page because a person just asked about it. The vendors' own documentation says so, and the bot names map to those jobs.
Treat them as three decisions:
- Training. Do you let the company learn from your pages?
- Search indexing. Do you want to be a source in its answers?
- User-triggered fetches. Can its assistant open your page when someone pastes a link or asks about you?
What does OpenAI's crawler documentation say?
OpenAI's crawler overview lists three user agents:
- GPTBot is for training. Disallow it and OpenAI is told not to use your content to train its foundation models.
- OAI-SearchBot controls search. Sites that opt out will not be shown in ChatGPT search answers, though they can still appear as plain navigational links.
- ChatGPT-User handles fetches that a user starts. OpenAI says it is not used for automatic crawling, and because a person triggers the request, robots.txt rules may not apply.
Two practical notes from the same docs: a robots.txt change can take around 24 hours to take effect for search, and the settings for GPTBot and OAI-SearchBot are independent. So "allow search, block training" is a supported setup.
What about Anthropic's ClaudeBot?
Anthropic's help center article on its crawlers describes three bots as well:
- ClaudeBot collects public web content that may feed model training.
- Claude-SearchBot analyzes pages to improve the relevance of search results. Blocking it can lower your visibility there.
- Claude-User fetches pages when someone asks Claude a question. Blocking it can reduce visibility in user-directed search.
Anthropic says its bots honor robots.txt, support the non-standard Crawl-delay directive, and do not try to bypass CAPTCHAs. Rules apply per hostname, so repeat them on every subdomain you care about.
And Perplexity?
Perplexity's crawler documentation recommends allowing PerplexityBot in robots.txt if you want your site to appear in its results, and says PerplexityBot is for surfacing sites in its search results, not for training foundation models. It also lists Perplexity-User, which fetches pages when a person asks a question. Because a user requested the fetch, Perplexity says this fetcher generally ignores robots.txt. Perplexity publishes IP ranges for both, which is handy if you filter traffic at the firewall.
Which setup fits a SaaS startup?
A summary of the three common choices:
| Goal | Allow | Block |
|---|---|---|
| Be cited in AI answers, keep content out of training | OAI-SearchBot, Claude-SearchBot, PerplexityBot | GPTBot, ClaudeBot |
| Maximum exposure | All of the above | Nothing |
| Private or paywalled content | Nothing on gated paths | Everything on those paths |
We'd pick the first row for most early-stage products. A young SaaS has far more to gain from being mentioned than from hiding a marketing site that is public anyway. The exception is a site whose value is the content itself, like a paid research library. Then block training and gate the good stuff.
A sample robots.txt
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
Check the current user-agent names in each vendor's docs before you paste this, because names and products change.
How do you check it worked?
- Open
yourdomain.com/robots.txtand confirm the file loads with a 200 status, not a redirect to a login page. - Wait a day. OpenAI says search can take around 24 hours to pick up a robots.txt change.
- Look at your server logs or CDN dashboard for the user-agent strings. If a bot you blocked keeps hitting you, compare its IP against the vendor's published list.
- Ask the assistants a question only your site can answer, and see whether they cite you.
Robots.txt is a request, not a lock. Anything you truly cannot afford to leak belongs behind a login. For a quick audit of the rest of your technical setup, try the free SEO tools on Woodander, and browse more guides in the SEO and AI visibility hub.
FAQ
Does blocking GPTBot remove me from ChatGPT search?
No. According to OpenAI's documentation, search visibility is controlled by OAI-SearchBot, and its settings are independent from GPTBot.
Will allowing AI search bots guarantee citations?
No. Access only makes you eligible. Clear pages, specific facts and a real audience still decide whether you get mentioned.
Do user-triggered bots obey robots.txt?
No, not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person starts the request. Perplexity says Perplexity-User generally ignores robots.txt for the same reason, so plan on gating sensitive pages instead of relying on a Disallow line.
How often should I review this file?
Every quarter is enough for most teams. Vendors add and rename bots, so skim their docs and your logs when you do.
Want people to find your product elsewhere while you sort this out? List it on Discover and see who else is shipping in your space.
