Does Your robots.txt Block AI Crawlers? How to Check in Two Minutes
If you are asking "does my robots.txt block AI crawlers", the answer is in the file itself: open yoursite.com/robots.txt in a browser and look for a User-agent: line naming an AI crawler (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended) followed by Disallow: /. That pair blocks it. If the file does not exist, or names none of them, every AI crawler may read your site.
Most sites that block an AI crawler did not decide to. A template, a security plugin or a "block AI" toggle did it for them, and the only sign is that the site never shows up in an assistant's answer. This post explains what each crawler does, how to read the file, and which lines to change.
Which AI crawlers matter, and what does each one do?
Each AI company runs separate crawlers for training, for search answers and for fetching a page a user asked about, and each is controlled by its own line in robots.txt. Blocking one does not block the others.
| Crawler | Company | What it does, per the company's docs |
|---|---|---|
| OAI-SearchBot | OpenAI | Finds pages for ChatGPT search answers. Opted-out sites are not shown in ChatGPT search. |
| GPTBot | OpenAI | Collects content for training. Independent of search. |
| ChatGPT-User | OpenAI | Fetches a page when a user asks ChatGPT to; robots.txt rules may not apply. |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity's results. Not used for training. |
| Perplexity-User | Perplexity | Fetches on a user's request; generally ignores robots.txt. |
| Claude-SearchBot | Anthropic | Improves the relevance of Claude's search responses. |
| ClaudeBot | Anthropic | Collects content that may contribute to training. |
| Claude-User | Anthropic | Fetches a page when a user asks Claude a question. |
| Google-Extended | A token controlling use for Gemini training and grounding. No ranking effect. |
The wording above is drawn from each company's own pages: OpenAI's crawler overview, Perplexity's crawler guide, Anthropic's help article (updated April 2026) and Google's list of common crawlers (updated October 2026). OpenAI's page says plainly that a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot, and that each setting is independent of the others. For what each OpenAI agent does, see what OAI-SearchBot is and whether to allow it; for Google's side, whether blocking Google-Extended hurts rankings.
How do you read a robots.txt file?
A robots.txt file is a list of groups. Each group starts with one or more User-agent: lines naming a crawler, followed by Disallow: and Allow: lines that apply to it. A crawler follows the group that names it most specifically; if no group names it, it follows the User-agent: * group; if there is no file at all, it may read everything. Those rules are not folklore: they are set out in RFC 9309, the Robots Exclusion Protocol the IETF published in 2022, which also says a crawler that gets a server error when fetching the file must treat the whole site as off limits until it can read it.
Three things to look for:
- A group that names an AI crawler with
Disallow: /. The slash on its own means the whole site. That crawler is out. - A
User-agent: *group withDisallow: /. Everyone is out, including Googlebot. This is rarer than it sounds but it happens on sites that copied a staging configuration. - A group that names the crawler with
Disallow:and nothing after it, orAllow: /. That crawler is in.
Changes are not instant. OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust, and Perplexity says the same, so a fix today shows up tomorrow.
How do sites block AI by accident?
Sites block AI by accident when a rule written for one purpose catches a crawler written for another. The common cases are a "block all AI bots" list that includes the search crawlers along with the training ones, a security setting that treats every unfamiliar user agent as a threat, and a rule added in 2023 against GPTBot before OAI-SearchBot existed, which then got extended by copy and paste.
The mistake underneath all three is treating "AI" as one thing. OpenAI, Perplexity and Anthropic each separate the crawler that feeds training from the crawler that feeds search answers, and Google's Google-Extended token does not impact a site's inclusion in Google Search, nor is it a ranking signal. A rule that blocks the training side and allows the search side is possible for every one of them. A rule that blocks both removes the site from the answers while doing nothing about Google.
Should you block training crawlers but allow search crawlers?
For most businesses, yes: allow the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) so your pages can be cited, and decide about the training crawlers (GPTBot, ClaudeBot, Google-Extended) on their own merits. Blocking training does not cost you a place in any assistant's answers; blocking search does. Our post on getting your website recommended by ChatGPT shows what the search side is worth.
The tradeoff on training is a judgement call about your content, not a search decision. What is not a judgement call is the search side: an assistant cannot cite a page its search crawler was told not to read.
The user-triggered fetchers (ChatGPT-User, Perplexity-User, Claude-User) are a third category. OpenAI and Perplexity both say those requests happen because a person asked, so robots.txt rules may not apply to them. They are not used to decide what appears in search, so there is little to gain from listing them.
Which lines do you change?
To allow the search crawlers and block the training crawlers, the groups look like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
To allow everything, delete any group that names an AI crawler, or give it Allow: /. Anthropic's article also notes that its bots honour Crawl-delay, so a site worried about load can slow ClaudeBot down rather than shut it out.
One caution: if a plugin or hosting panel manages the file for you, edit it there rather than by hand, or the next save will overwrite your change.
How do you know it worked?
You know it worked when the live file shows the lines you intended, a day has passed, and the assistant's answer to a question your page covers lists your site among its sources. Check the file by loading it in a browser, not by looking at the editor, because caches and plugins can serve a different version from the one you saved.
PARCHMNT is a writing tool that researches what ranks for a search, writes the page, checks every claim and tracks where it lands in Google and AI answers. It reads each customer's robots.txt the way each crawler would, once a week, and names any assistant a rule shuts out, with the line to remove. The check is code, not AI. When we wrote it, we had to read all four companies' pages side by side to get the names and the rules right; the table above is that comparison, so you don't have to.
Key takeaways
- Open yoursite.com/robots.txt and look for a
User-agent:line naming an AI crawler followed byDisallow: /. - Search crawlers and training crawlers are separate settings at OpenAI, Perplexity, Anthropic and Google.
- Blocking GPTBot, ClaudeBot or Google-Extended does not remove a site from AI search answers; blocking OAI-SearchBot, PerplexityBot or Claude-SearchBot does.
- Google-Extended has no effect on Google Search ranking.
- Changes take about a day to be reflected at OpenAI and Perplexity.
Frequently asked questions
Does blocking GPTBot remove a site from ChatGPT search?
No. OpenAI's documentation says GPTBot controls training and OAI-SearchBot controls ChatGPT search, and that each setting is independent. A site can block GPTBot and still appear in ChatGPT search answers, as long as OAI-SearchBot is allowed.
Does Google-Extended affect AI Overviews?
Google says Google-Extended does not affect a site's inclusion in Search and is not a ranking signal. Its documentation on controlling what you share points to the robots meta tags and a Search Console control for managing inclusion in AI Overviews and AI Mode, which are separate from the Google-Extended token.
Can the user-triggered fetchers be blocked?
OpenAI says robots.txt rules may not apply to ChatGPT-User because the request is initiated by a user, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason. Neither is used to decide what appears in search results, so blocking them changes little.
What if a site has no robots.txt at all?
Then every crawler may read it. A missing file is an open door, which is fine for most sites. It only becomes a problem when someone later adds a file from a template without reading it.
Does my robots.txt block AI crawlers? The two-minute check, once a quarter
Load the file, read the groups, fix the lines, wait a day. If you would rather have it checked for you every week, try one article free and the check comes with it.
Sources
This post was written and checked with the same rules PARCHMNT applies to every page it writes for a customer: researched against what ranks today, answer first, every figure with a source.
Try 1 article free