How-tos · November 11, 2026 · 10 min read
GPTBot vs OAI-SearchBot: Which OpenAI Crawlers to Allow
GPTBot crawls for model training. OAI-SearchBot crawls for ChatGPT search. What each OpenAI bot does, its user agent, robots.txt rules and how to verify it.
GPTBot and OAI-SearchBot are two separate OpenAI crawlers with two separate jobs. GPTBot crawls content that may be used to train OpenAI’s foundation models. OAI-SearchBot crawls pages so they can appear in ChatGPT’s search answers. You can allow one and block the other in robots.txt, and OpenAI treats each setting on its own.
So the usual choice is simple. If you want your pages to appear in ChatGPT search, allow OAI-SearchBot. Whether you allow GPTBot is a separate decision about training.
Every fact about OpenAI’s crawlers in this post comes from OpenAI’s crawler overview, last checked 1 November 2026. OpenAI can change its crawlers and their names. That page is the source of truth.
The OpenAI crawlers at a glance
OpenAI’s page lists four user agents. Two of them, GPTBot and OAI-SearchBot, are the ones the page names as robots.txt controls.
| User agent | What it does, per OpenAI | Automatic crawler? | robots.txt effect |
|---|---|---|---|
| OAI-SearchBot | Surfaces websites in search results in ChatGPT’s search features | Yes | Opted-out sites are not shown in ChatGPT search answers, but can still appear as navigational links |
| GPTBot | Crawls content that may be used to train OpenAI’s generative AI foundation models | Yes | Disallowing it signals that your content should not be used for training |
| ChatGPT-User | Visits a page when a user’s question in ChatGPT or a Custom GPT calls for it, and handles GPT Actions | No | Robots.txt rules may not apply, because a user starts the visit |
| OAI-AdsBot | Checks the safety of landing pages submitted as ChatGPT ads, and may use them to judge ad relevance | No, it visits only pages submitted as ads | OpenAI’s page gives no robots.txt guidance for it. It says the data is not used to train foundation models |
Three points from the same page matter for most sites:
- The settings are independent. OpenAI gives the example of a site that allows OAI-SearchBot to appear in search and disallows GPTBot to opt out of training.
- One crawl can serve both. If you allow both bots, OpenAI says it may use the results of one crawl for both purposes, to avoid crawling twice.
- Search changes take about a day. OpenAI says it can take about 24 hours after a robots.txt update for its search systems to adjust.
What is GPTBot?
GPTBot is OpenAI’s training crawler. OpenAI says it is used to make its generative AI foundation models “more useful and safe”, and it crawls content that may be used in training them. If you disallow GPTBot in robots.txt, you tell OpenAI not to use your content for training.
The GPTBot user agent, as an example from OpenAI’s page (the version number may change):
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Blocking GPTBot does not remove you from ChatGPT search answers. Search uses OAI-SearchBot, and the two settings are separate.
What is OAI-SearchBot?
OAI-SearchBot is OpenAI’s search crawler. OpenAI says it is used to surface websites in ChatGPT’s search features. OpenAI recommends that you allow OAI-SearchBot in robots.txt, and allow requests from its published IP ranges, if you want your site to appear in search results.
The OAI-SearchBot user agent, as an example from OpenAI’s page (the version number may change):
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
When GPTBot or OAI-SearchBot fetches your robots.txt file, the user agent may also contain a robots.txt marker. OpenAI adds it so you can tell robots.txt requests apart from other requests in your logs.
What is ChatGPT-User?
ChatGPT-User is not a crawler in the usual sense. It visits a page when a person asks ChatGPT or a Custom GPT a question and the answer needs that page. It also handles GPT Actions. OpenAI says ChatGPT-User does not crawl the web automatically, and it is not used to decide whether content can appear in search. Because a user starts each visit, OpenAI says robots.txt rules may not apply.
The full ChatGPT-User user agent from OpenAI’s page:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
What is OAI-AdsBot?
OAI-AdsBot only visits landing pages that an advertiser submits as ChatGPT ads. It checks that the page meets OpenAI’s ad policies, and OpenAI may use the page content to decide when the ad is relevant. It visits only pages submitted as ads, so most sites never see it. Its full user agent:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot
What blocking each one does
| You block | Effect on ChatGPT search answers | Effect on training |
|---|---|---|
| GPTBot only | None. OAI-SearchBot still crawls for search. | You signal that your content should not be used for training. |
| OAI-SearchBot only | Your pages are not shown in ChatGPT search answers. They can still appear as navigational links. | None. GPTBot still crawls, if you allow it. |
| Both | Not shown in search answers, except as navigational links. | You signal no training use. |
| ChatGPT-User | Not used to decide what appears in search. If a block is honored, ChatGPT may not open your page for a user’s question. OpenAI says robots.txt rules may not apply to it. | OpenAI names GPTBot, not ChatGPT-User, as the training control. |
robots.txt examples
Put these rules in the robots.txt file at the root of your domain. Each User-agent line names the bot. Each Disallow or Allow line sets what it may crawl.
Allow ChatGPT search, block training
This is the most common choice for a site that wants to appear in ChatGPT search answers but does not want its content used for training.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
Allow both
If you are fine with both uses, you do not need rules for either bot. A robots.txt with no rule for them allows them. To make the choice explicit:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Allow: /
Block training for one folder only
You can block GPTBot from part of the site and allow the rest.
User-agent: GPTBot
Disallow: /members/
Block OpenAI’s crawlers and ChatGPT-User
This blocks both automatic crawlers. It also adds a rule for ChatGPT-User, but OpenAI says robots.txt rules may not apply to user-initiated visits, so treat that line as a request, not a guarantee.
User-agent: OAI-SearchBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
After you change robots.txt, allow about 24 hours for ChatGPT search to adjust. Also check your firewall or CDN bot settings. A rule there can block OAI-SearchBot even when robots.txt allows it.
An llms.txt file is a different thing. It points AI tools to your content. It does not allow or block crawlers. Robots.txt is the file OpenAI’s page names for these controls.
How to verify that a bot is really OpenAI
Anyone can send a request with “GPTBot” in the user agent. The user agent alone proves nothing. OpenAI publishes the IP ranges of each bot as a JSON file. Each file lists IPv4 address ranges in CIDR form.
| User agent | Published IP ranges |
|---|---|
| OAI-SearchBot | openai.com/searchbot.json |
| GPTBot | openai.com/gptbot.json |
| ChatGPT-User | openai.com/chatgpt-user.json |
| OAI-AdsBot | openai.com/adsbot.json |
To check a request:
- Read the user agent in your server or CDN log. Note which bot it claims to be.
- Open the JSON file for that bot.
- Check whether the request’s IP address falls inside one of the listed ranges.
- If it does not, the request did not come from that OpenAI bot. Treat it as an ordinary unknown visitor.
The ranges change over time. Each file has a creation time at the top. If you allowlist these ranges in a firewall, refresh them on a schedule instead of copying them once.
Why this matters for AI visibility
ChatGPT answers in two ways. Sometimes it answers from what the model learned in training. Sometimes it searches the web and cites the pages it reads. Our guide to ChatGPT SEO explains both modes.
The two crawlers map to those two modes:
- OAI-SearchBot feeds the search mode. If it cannot crawl your pages, OpenAI says they are not shown in ChatGPT search answers. A page that is not shown cannot be cited as a source. For a buyer question such as “best X for Y”, that can mean a rival’s page is cited and yours is not.
- GPTBot feeds future training. Blocking it affects what future models may learn from your site. It does not change search answers today.
So if your goal is to be named and cited in ChatGPT, check OAI-SearchBot first. Our post on where ChatGPT gets its citations covers which sources it tends to cite.
Robots.txt controls access. It does not tell you whether ChatGPT actually names or cites you. To know that, you sample the answers. CitePrism’s ChatGPT brand monitoring runs your questions against ChatGPT and six other engines on a schedule and stores every raw answer. AI citation tracking shows which domains each answer cites, so you can see whether your pages appear after a robots.txt change. CitePrism does not read your server logs or show bot traffic. For that, use your own logs or your CDN.
FAQ
What is GPTBot?
GPTBot is OpenAI’s web crawler for model training. OpenAI says it crawls content that may be used to train its generative AI foundation models. If you disallow GPTBot in robots.txt, you signal that your content should not be used for training. Blocking it does not remove your pages from ChatGPT search answers. Last checked 1 November 2026.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot crawls for training. OAI-SearchBot crawls for ChatGPT search. OpenAI treats the two robots.txt settings as independent, so you can allow search and block training. If you allow both, OpenAI says it may use one crawl for both purposes.
Does blocking GPTBot remove my site from ChatGPT?
Not from ChatGPT search answers. Search depends on OAI-SearchBot, not GPTBot. If you block OAI-SearchBot, OpenAI says your pages are not shown in ChatGPT search answers, though they can still appear as navigational links.
Should I block GPTBot?
It depends on whether you want your content used for model training. That is a business and legal choice, not a visibility tactic. Blocking GPTBot does not change ChatGPT search answers today. If you want to appear in ChatGPT search, keep OAI-SearchBot allowed whatever you decide about GPTBot.
What is ChatGPT-User?
ChatGPT-User is the user agent ChatGPT uses when a person’s question needs it to visit a page, and for GPT Actions. It does not crawl automatically, and OpenAI says it is not used to decide what appears in search. Because a user starts each visit, OpenAI says robots.txt rules may not apply to it.
How long does a robots.txt change take to affect ChatGPT search?
OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust. Training effects have no published timeline.
How do I know a GPTBot request is real?
Check the request’s IP address against the range list OpenAI publishes at openai.com/gptbot.json. The other bots have their own files. A request with the right user agent but an IP address outside the list did not come from OpenAI.