What Is OAI-SearchBot and Should You Allow It? Yes, Here Is Why
OAI-SearchBot is OpenAI's search crawler, the one that decides whether ChatGPT search can show your site, and you should allow it unless you want to be absent from ChatGPT's answers. It is separate from GPTBot, which gathers training data, and from ChatGPT-User, which fetches a page when a person asks. Allowing search while blocking training is a two-line change.
Everything below comes from OpenAI's own crawler page, rather than the plugin vendor blogs that hold the search results. That page describes the three agents, their user agent strings, their IP ranges and how long a robots.txt change takes to apply. We add RFC 9309, the robots.txt standard, and the way we check the result in ChatGPT itself.
What is OAI-SearchBot?
OAI-SearchBot is the crawler OpenAI uses to find pages for ChatGPT's search features, the role Googlebot plays for Google. OpenAI's documentation says it "is used to surface websites in search results in ChatGPT's search features". Sites that opt out "will not be shown in ChatGPT search answers, though can still appear as navigational links".
The name tells you its job. OpenAI runs one agent for search, one for training and one for user requests, and gives each its own robots.txt token. It recommends allowing this one. "To help ensure your site appears in search results, we recommend allowing OAI-SearchBot in your site's robots.txt file and allowing requests from our published IP ranges below," the page says.
Its user agent identifies itself plainly. The example string OpenAI publishes ends in compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot and starts like a desktop Chrome browser. Search your logs for OAI-SearchBot to see whether the ChatGPT search crawler has visited. To be sure a visit is genuine, check the IP, as described below.
What are the three OpenAI bots and what does each one do?
OpenAI runs three agents: OAI-SearchBot for search, GPTBot for training, and ChatGPT-User for actions a person triggers inside ChatGPT. Each has its own robots.txt token. OpenAI says "each setting is independent of the others", so you can allow search, block training and accept that user requests may not follow robots.txt at all.
| Agent | What OpenAI says it does | Follows robots.txt? | Allow it if you want |
|---|---|---|---|
| OAI-SearchBot | Surfaces websites "in ChatGPT's search features" | Yes; it is the token for "Search opt outs and automatic crawl" | Your pages cited in ChatGPT search |
| GPTBot | Crawls content "that may be used in training" OpenAI's models | Yes; disallowing it says your content "should not be used in training" | Your content used to improve OpenAI's models |
| ChatGPT-User | "Certain user actions in ChatGPT and Custom GPTs"; no automatic crawling | Not reliably; rules "may not apply" because a person asked | People to open your pages from inside ChatGPT |
Two details matter in practice. OpenAI says that if both bots are allowed, "we may use the results from just one crawl for both use cases". So the way to say no to training is a GPTBot rule, not the absence of a search rule; to block GPTBot, name it. And ChatGPT-User "is not used to determine whether content may appear in Search", so blocking it does nothing for your search presence. OpenAI adds that ChatGPT users may also reach external applications through GPT Actions. All three agents publish IP ranges at openai.com/searchbot.json, openai.com/gptbot.json and openai.com/chatgpt-user.json.
Should you allow OAI-SearchBot?
Yes, for almost every business and publisher. Allow OAI-SearchBot if you want ChatGPT to cite, link or recommend your pages, because OpenAI says opted-out sites are not shown in ChatGPT search answers. The only reason to block it is that you do not want to appear in ChatGPT at all.
The tradeoff people worry about is training, and OpenAI separates it. Allowing OAI-SearchBot does not change your training setting; GPTBot does. A site that allows OAI-SearchBot and disallows GPTBot follows OpenAI's own example. A webmaster, it says, "can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training".
Who should still think twice: sites with pages that should not be summarized out of context, such as paywalled, member-only or regulated content. For those pages, the answer is a Disallow on their paths or a login, not a site-wide block. Our guide to getting your website recommended by ChatGPT covers what happens once the crawler is allowed in.
We treat the search crawler as a visibility setting and the training crawler as a policy setting, and we recommend deciding them separately.
Which robots.txt lines allow OAI-SearchBot and block GPTBot?
Two groups do it: the robots.txt group for OAI-SearchBot with Allow: /, and a group for GPTBot with Disallow: /. Each group starts with its own User-agent: line. The order between groups does not matter, because a crawler reads the group that names it.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
Three rules from RFC 9309, the Robots Exclusion Protocol published by the IETF in September 2022, decide how those lines are read:
- The product token match is case-insensitive, so
oai-searchbotworks as well asOAI-SearchBot. - A crawler with no group of its own obeys the
User-agent: *group. If your file ends withUser-agent: *andDisallow: /, every unnamed crawler is out. - If the server returns a 5xx error for robots.txt, the crawler "MUST assume complete disallow". A robots.txt that errors blocks everything until it loads again.
Then the timing. OpenAI notes that "it can take ~24 hours from a site's robots.txt update for our systems to adjust", so a fix today shows tomorrow. Load the live file in a browser after saving, because a plugin or cache can serve an older version. Our two-minute robots.txt check lists the other companies' tokens.
How do you verify that a visitor is really OAI-SearchBot?
Check the visitor's IP address against OpenAI's published list at openai.com/searchbot.json, because a user agent string can be typed by anyone. A request that calls itself OAI-SearchBot but comes from outside OAI-SearchBot's IP ranges is not OpenAI, whatever the string says.
The steps:
- Pull the requests whose user agent contains
OAI-SearchBotfrom your server or CDN logs. - Download
https://openai.com/searchbot.json. It lists IPv4 prefixes under aprefixeskey with acreationTimestamp, and held 39 prefixes when we opened it in October 2026. Fetch it fresh rather than copying it into a rule. - Check each request's IP against those prefixes. Matches are genuine; the rest are imposters you can block without touching ChatGPT.
The check matters because blocking "OAI-SearchBot" by user agent in a firewall, as some bot-protection rules do, blocks the real crawler along with the fakes. OpenAI asks sites to allow requests "from our published IP ranges", and an allow rule by IP does that without opening the door to everything.
How do you check whether ChatGPT search can see your site?
Ask ChatGPT, with search turned on, a question your page answers, and open the sources panel beneath the answer to see whether your page is listed. Use two or three wordings, note whether you are cited and whether you are named, and repeat monthly, because answers change as the web does.
If your site never appears, check the robots.txt group first, wait the day OpenAI asks for, then look at what was cited instead. Being crawlable is the floor; the pages that get cited answer the question in their first paragraph and link their sources. In our experience the file is the first thing to check, because a blocking rule fails silently: no error, no notice, just absence.
PARCHMNT is a writing tool that researches what ranks for a search, writes the page, checks every claim and tracks where it lands in Google and AI answers. Its crawler check reads a site's robots.txt weekly the way OAI-SearchBot, GPTBot and ChatGPT-User each would. Its monthly visibility check asks ChatGPT the article's questions and records whether the site was cited, named or recommended.
Key takeaways
- OAI-SearchBot is OpenAI's search crawler; sites that block it "will not be shown in ChatGPT search answers", by OpenAI's account.
- GPTBot controls training and ChatGPT-User handles a person's requests; OpenAI says each setting is independent.
- Allow OAI-SearchBot and, if you choose, disallow GPTBot: two groups in robots.txt, read case-insensitively under RFC 9309.
- Verify real visits against
openai.com/searchbot.json, not against the user agent string alone. - OpenAI says a robots.txt change takes about 24 hours to apply for search; check the sources panel in ChatGPT after that.
Frequently asked questions
Does OAI-SearchBot render JavaScript?
OpenAI's crawler documentation does not say, as of October 2026, so any claim either way is a guess. The safe practice is the same as for every crawler. Put the text you want cited in the HTML the server sends, and keep anything that only appears after scripts run to extras rather than the answer itself.
Does blocking OAI-SearchBot stop ChatGPT-User?
No, it does not. OpenAI says ChatGPT-User handles actions a person starts inside ChatGPT or a Custom GPT, and that robots.txt rules may not apply to it. The same page says it "is not used to determine whether content may appear in Search". Blocking OAI-SearchBot removes you from ChatGPT search answers. It does not stop a person asking ChatGPT to read a page they already have the address of.
Can a blocked site still appear as a navigational link in ChatGPT?
Yes. OpenAI's wording is that opted-out sites "will not be shown in ChatGPT search answers, though can still appear as navigational links". That reads as: ChatGPT may still link to your site when the person is plainly looking for it, while your pages stop being used as sources for answers. Blocking the crawler costs citations, not existence.
Does allowing OAI-SearchBot mean OpenAI trains on your pages?
Not on its own. OpenAI ties training to GPTBot: disallowing GPTBot "indicates a site's content should not be used in training generative AI foundation models". Its documentation also says that when both bots are allowed, one crawl may serve both uses. So the training decision is the GPTBot line. Allow search, disallow training, and you have followed OpenAI's own example.
What is OAI-SearchBot and should you allow it? In one paragraph
OAI-SearchBot is the crawler that puts your pages in front of ChatGPT search, and you should allow it unless you want to be invisible there. Decide training separately with GPTBot, verify real visits by IP, wait a day after any change, and then ask ChatGPT your own questions. If you would rather have the file checked weekly and the asking done monthly, try one article free.
Sources
This post was written and checked with the same rules PARCHMNT applies to every page it writes for a customer: researched against what ranks today, answer first, every figure with a source.
Try 1 article free