Free report
Menu Close

Should you block AI crawlers?

AI companies run two kinds of crawler: ones that collect pages to train models, and ones that fetch pages so an assistant can cite them in an answer. Blocking the first kind keeps your content out of training. Blocking the second kind removes your business from AI search. A blanket block does both, which is rarely what a business wants.

Updated

This guide, surveyed at release eab6db0 on 19 October 2026: words per section.

Each sounding opens its section

  1. 01 Two kinds of crawler 165 words
  2. 02 What a business should allow 102 words
  3. 03 The robots.txt lines 50 words
  4. 04 Check what your site does now 93 words
  5. 05 What blocking does not do 54 words
  6. 06 Questions people ask 118 words
  7. 07 Free competitor gap report 107 words
  8. 08 More in this series 80 words
Words
769
Site 366–1,117 · median 749.5
Sections (H2)
8
Site 3–14 · median 9
Internal links
14
Site 1–36 · median 19
File
71.5 kB
HTML, uncompressed · of 83.6 kB budget · 0 scripts

Sheet

078 of 126

Release eab6db0

Standard 49 · 101 checks

Two kinds of crawler

The AI companies listed below each separate the crawler that gathers training material from the ones that serve search. The names differ, and the jobs do not.

Crawler Company What it does
GPTBot OpenAI Collects content that may be used to train models
OAI-SearchBot OpenAI Finds pages to show in ChatGPT search
ChatGPT-User OpenAI Fetches a page when a person’s request needs it
ClaudeBot Anthropic Collects content that may be used to train models
Claude-SearchBot Anthropic Indexes pages for search results in Claude
Claude-User Anthropic Fetches a page when a person asks Claude a question
PerplexityBot Perplexity Finds pages to show and link in Perplexity’s results
Perplexity-User Perplexity Fetches a page when a person’s question needs it

Google works differently. Its AI features in Search use the same crawling as ordinary Google Search. A separate robots.txt name, Google-Extended, controls whether pages are used for training and grounding in Google’s other AI systems, and Google says it does not affect Search.

What a business should allow

A business that wants to be found has a clear interest in the search crawlers and the user fetchers: they are how an assistant reads the site and links to it. Blocking them removes the business from answers its competitors will appear in.

Training crawlers are a different decision. Allowing them means your pages may shape what future models know. Blocking them keeps the content out, at no cost to search visibility. For a site whose pages exist to win customers, allowing them is the common choice; for a publisher whose content is the product, blocking is.

The robots.txt lines

A robots.txt file with no AI rules allows every crawler. To block training and keep search, name the training crawlers only.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Leave OAI-SearchBot, Claude-SearchBot and PerplexityBot unblocked, or the site drops out of those assistants’ answers.

Check what your site does now

Three places decide what a crawler is allowed to do.

  • The robots.txt file. Open it at your domain followed by /robots.txt and read every Disallow line.
  • The platform. Some website builders have a setting for AI crawlers.
  • The host or security service. Some can block AI crawlers before robots.txt is read, and a block there overrides everything else.

This site allows every crawler in the table. Its robots.txt has one group, for all crawlers, and it disallows only addresses that hold no content.

What blocking does not do

A robots.txt rule is a request. The named companies say their crawlers follow it, but it does not remove what was collected before the rule existed, and it does not bind crawlers that ignore the convention. For what happens after the crawl, see how ChatGPT search chooses websites.

Questions people ask

What is GPTBot?

GPTBot is OpenAI's crawler for collecting content that may be used to train its models. Blocking it does not remove a site from ChatGPT search, which uses a different crawler, OAI-SearchBot.

Does blocking AI crawlers hurt SEO?

Blocking training crawlers does not change how Google ranks a site. Blocking search crawlers such as OAI-SearchBot or PerplexityBot removes the site from those assistants' answers, which is a loss of visibility there.

Do AI crawlers obey robots.txt?

OpenAI, Anthropic and Perplexity state that their indexing and training crawlers follow robots.txt. Fetches made at a user's direct request are treated differently: OpenAI and Perplexity both say robots.txt may not apply to those.

Free competitor gap report

Whether assistants can read a site is one line in a larger comparison. Name up to three competitors and we check what your site and theirs allow, and how each reads to an answer engine, then send you where they are easier to find.

  1. Speed and technical health, side by side
  2. On-page SEO and site structure
  3. The content they are found for and you are not
  4. Who holds the searches that matter in your market
  5. A prioritised list of what to fix first
Request your free report by email
Address
contact@hadalmedusa.com
Include
your site; up to three competitors; where you compete

More in this series