> Which AI crawlers collect training data and which make a site appear in AI search, the robots.txt lines for each, and what a business should allow. - Canonical: https://hadalmedusa.com/guides/should-you-block-ai-crawlers/ --- # Should you block AI crawlers? AI companies run two kinds of crawler: ones that collect pages to train models, and ones that fetch pages so an assistant can cite them in an answer. Blocking the first kind keeps your content out of training. Blocking the second kind removes your business from AI search. A blanket block does both, which is rarely what a business wants. Updated 6 October 2026 ## Two kinds of crawler The AI companies listed below each separate the crawler that gathers training material from the ones that serve search. The names differ, and the jobs do not. Crawler | Company | What it does | `GPTBot` | OpenAI | Collects content that may be used to train models | `OAI-SearchBot` | OpenAI | Finds pages to show in ChatGPT search | `ChatGPT-User` | OpenAI | Fetches a page when a person’s request needs it | `ClaudeBot` | Anthropic | Collects content that may be used to train models | `Claude-SearchBot` | Anthropic | Indexes pages for search results in Claude | `Claude-User` | Anthropic | Fetches a page when a person asks Claude a question | `PerplexityBot` | Perplexity | Finds pages to show and link in Perplexity’s results | `Perplexity-User` | Perplexity | Fetches a page when a person’s question needs it | Google works differently. Its AI features in Search use the same crawling as ordinary Google Search. A separate `robots.txt` name, `Google-Extended`, controls whether pages are used for training and grounding in Google’s other AI systems, and Google says it does not affect Search. ## What a business should allow A business that wants to be found has a clear interest in the search crawlers and the user fetchers: they are how an assistant reads the site and links to it. Blocking them removes the business from answers its competitors will appear in. Training crawlers are a different decision. Allowing them means your pages may shape what future models know. Blocking them keeps the content out, at no cost to search visibility. For a site whose pages exist to win customers, allowing them is the common choice; for a publisher whose content is the product, blocking is. ## The robots.txt lines A `robots.txt` file with no AI rules allows every crawler. To block training and keep search, name the training crawlers only. ``` User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / ``` Leave `OAI-SearchBot`, `Claude-SearchBot` and `PerplexityBot` unblocked, or the site drops out of those assistants’ answers. ## Check what your site does now Three places decide what a crawler is allowed to do. - The `robots.txt` file. Open it at your domain followed by `/robots.txt` and read every `Disallow` line. - The platform. Some website builders have a setting for AI crawlers. - The host or security service. Some can block AI crawlers before `robots.txt` is read, and a block there overrides everything else. This site allows every crawler in the table. Its `robots.txt` has one group, for all crawlers, and it disallows only addresses that hold no content. ## What blocking does not do A `robots.txt` rule is a request. The named companies say their crawlers follow it, but it does not remove what was collected before the rule existed, and it does not bind crawlers that ignore the convention. For what happens after the crawl, see [how ChatGPT search chooses websites](https://hadalmedusa.com/guides/how-chatgpt-search-chooses-websites/). ## Questions people ask ### What is GPTBot? GPTBot is OpenAI's crawler for collecting content that may be used to train its models. Blocking it does not remove a site from ChatGPT search, which uses a different crawler, OAI-SearchBot. ### Does blocking AI crawlers hurt SEO? Blocking training crawlers does not change how Google ranks a site. Blocking search crawlers such as OAI-SearchBot or PerplexityBot removes the site from those assistants' answers, which is a loss of visibility there. ### Do AI crawlers obey robots.txt? OpenAI, Anthropic and Perplexity state that their indexing and training crawlers follow robots.txt. Fetches made at a user's direct request are treated differently: OpenAI and Perplexity both say robots.txt may not apply to those. ## Free competitor gap report Whether assistants can read a site is one line in a larger comparison. Name up to three competitors and we check what your site and theirs allow, and how each reads to an answer engine, then send you where they are easier to find. 1. Speed and technical health, side by side 2. On-page SEO and site structure 3. The content they are found for and you are not 4. Who holds the searches that matter in your market 5. A prioritised list of what to fix first [Request your free report by email](mailto:contact@hadalmedusa.com?subject=Competitor%20gap%20report%3A%20Guide%3A%20Should%20you%20block%20AI%20crawlers%3F&body=Website%3A%0D%0A%0D%0AWhere%20you%20compete%20(area%2C%20country%20or%20countries)%3A%0D%0A%0D%0ACompetitors%20to%20compare%20against%20(up%20to%20three)%3A%0D%0A%0D%0AAnything%20else%20we%20should%20know%3A%0D%0A%0D%0A) - **Address** — contact@hadalmedusa.com - **Include** — your site; up to three competitors; where you compete ## More in this series - [How to check if ChatGPT recommends your business](https://hadalmedusa.com/guides/does-chatgpt-recommend-your-business/) - [How to get ChatGPT to recommend your business](https://hadalmedusa.com/guides/how-to-get-chatgpt-to-recommend-your-business/) - [How ChatGPT search chooses the websites it cites](https://hadalmedusa.com/guides/how-chatgpt-search-chooses-websites/) - [What llms.txt is, with an example](https://hadalmedusa.com/guides/what-is-llms-txt/) - [How to see ChatGPT traffic in Google Analytics](https://hadalmedusa.com/guides/chatgpt-traffic-in-google-analytics/) - [AI Overviews and falling clicks: how to check, and what to do](https://hadalmedusa.com/guides/ai-overviews-traffic-drop/) - [Generative engine optimisation: what it means and what works](https://hadalmedusa.com/guides/generative-engine-optimisation/) - [AI search and answer engines: the whole topic](https://hadalmedusa.com/topics/ai-search/) - [All guides](https://hadalmedusa.com/guides/) - [On-site SEO: what the service includes](https://hadalmedusa.com/services/seo/)