> What 'Blocked by robots.txt' means in Google Search Console, when the block is intended, how to find the rule, and why it is the wrong way to hide a page. - Canonical: https://hadalmedusa.com/guides/blocked-by-robots-txt/ --- # Blocked by robots.txt: what it means and how to fix it 'Blocked by robots.txt' means a rule in the site's robots.txt file told Google not to fetch the page, so Google has not read it. The file sits at the root of the site and applies to whole paths at once, which makes it easy for one line to block far more than was meant. Updated 6 October 2026 ## What the status means The `robots.txt` file lists parts of a site that crawlers are asked not to fetch. Google’s Page indexing report puts a page under this status when a rule in that file matched its address. Google did not read the page, so it knows the address and nothing about the content. ## When the block is intended Blocking is right for parts of a site with no value in search and many addresses: baskets, checkouts, account areas, internal search results. Hosted platforms block these by default. A list of such addresses needs no action. ## When it is a mistake The status needs attention when it lists pages that should be found. Three causes are common. - A rule left from when the site was being built, often `Disallow: /`, which blocks everything. - A rule written for one folder that also matches others, because rules match the start of an address. - A plugin or app that added rules of its own. ## Find the rule Open your domain followed by `/robots.txt` in a browser. Read each `Disallow` line under `User-agent: *` and under any group naming Googlebot. The rule responsible is the one whose path begins the same way as the blocked address. Search Console’s robots.txt report shows the file as Google last fetched it. ## Fix it Remove the rule, or narrow it so that it covers only what should be blocked. Where the file is edited depends on the platform: directly on the server, through an SEO plugin on WordPress, or in the platform’s settings on Wix and Shopify. Then request indexing for the pages that matter. ## Why it is the wrong way to hide a page A blocked page can still turn up in search. Google’s guidance says so plainly: to make sure a page is not indexed, remove the robots.txt block and use a `noindex` instruction instead. A block stops Google reading the page, and so stops it seeing the instruction that would keep the page out. The companion status is covered in [excluded by noindex tag](https://hadalmedusa.com/guides/excluded-by-noindex-tag/), and the choices about AI crawlers in [should you block AI crawlers?](https://hadalmedusa.com/guides/should-you-block-ai-crawlers/) ## Questions people ask ### How do I fix 'Blocked by robots.txt'? Open the robots.txt file, find the Disallow rule that matches the page's address, and remove or narrow it. On hosted platforms the file is edited through the platform's settings, where the platform allows it. ### Should I use robots.txt to keep a page out of Google? No. Google's guidance for this status is that a page blocked by robots.txt can still be indexed if other pages link to it. To keep a page out of search, allow it to be fetched and give it a noindex instruction. ### What does 'Indexed, though blocked by robots.txt' mean? The page is in Google's index even though Google was not allowed to read it, usually because other pages link to it. It appears in results with little or no description. ## Free competitor gap report One line in one file can hide a whole section of a site. Name up to three competitors and we compare what your site and theirs let search engines read, then send you what yours is keeping from them. 1. Speed and technical health, side by side 2. On-page SEO and site structure 3. The content they are found for and you are not 4. Who holds the searches that matter in your market 5. A prioritised list of what to fix first [Request your free report by email](mailto:contact@hadalmedusa.com?subject=Competitor%20gap%20report%3A%20Guide%3A%20Blocked%20by%20robots.txt%3A%20what%20it%20means%20and%20how%20to%20fix%20it&body=Website%3A%0D%0A%0D%0AWhere%20you%20compete%20(area%2C%20country%20or%20countries)%3A%0D%0A%0D%0ACompetitors%20to%20compare%20against%20(up%20to%20three)%3A%0D%0A%0D%0AAnything%20else%20we%20should%20know%3A%0D%0A%0D%0A) - **Address** — contact@hadalmedusa.com - **Include** — your site; up to three competitors; where you compete ## More in this series - [Why your website is not showing up on Google](https://hadalmedusa.com/guides/website-not-showing-on-google/) - [Why your Wix site is not showing up on Google](https://hadalmedusa.com/guides/wix-site-not-showing-on-google/) - [Crawled, currently not indexed: what it means and how to fix it](https://hadalmedusa.com/guides/crawled-currently-not-indexed/) - [Discovered, currently not indexed: what it means and how to fix it](https://hadalmedusa.com/guides/discovered-currently-not-indexed/) - [Page with redirect: what it means in Search Console](https://hadalmedusa.com/guides/page-with-redirect-search-console/) - [Why your Squarespace site is not showing up on Google](https://hadalmedusa.com/guides/squarespace-site-not-showing-on-google/) - [Why your Shopify store is not showing up on Google](https://hadalmedusa.com/guides/shopify-store-not-showing-on-google/) - [Why your Google Business Profile is not showing up](https://hadalmedusa.com/guides/google-business-profile-not-showing-up/) - [Why your Shopify store is not getting traffic or sales](https://hadalmedusa.com/guides/shopify-store-not-getting-traffic/) - [Why your WordPress site is not showing up on Google](https://hadalmedusa.com/guides/wordpress-site-not-showing-on-google/) - [Alternate page with proper canonical tag: what it means](https://hadalmedusa.com/guides/alternate-page-with-proper-canonical-tag/) - [Duplicate without user-selected canonical: what it means and how to fix it](https://hadalmedusa.com/guides/duplicate-without-user-selected-canonical/) - [Duplicate, Google chose different canonical than user: what to do](https://hadalmedusa.com/guides/duplicate-google-chose-different-canonical/) - [Excluded by noindex tag: what it means and how to fix it](https://hadalmedusa.com/guides/excluded-by-noindex-tag/) - [Soft 404 and Not found (404): what they mean and how to fix them](https://hadalmedusa.com/guides/soft-404/) - [Server error (5xx): what it means and how to fix it](https://hadalmedusa.com/guides/server-error-5xx-search-console/) - [Redirect error in Search Console: what it means and how to fix it](https://hadalmedusa.com/guides/redirect-error-search-console/) - [Indexed, though blocked by robots.txt: what it means and what to do](https://hadalmedusa.com/guides/indexed-though-blocked-by-robots-txt/) - [Blocked due to access forbidden (403): what it means and how to fix it](https://hadalmedusa.com/guides/blocked-due-to-access-forbidden-403/) - [How to make your website show up on Google](https://hadalmedusa.com/guides/how-to-make-your-website-show-up-on-google/) - [When a site is not showing on Google: the whole topic](https://hadalmedusa.com/topics/not-on-google/) - [All guides](https://hadalmedusa.com/guides/) - [On-site SEO: what the service includes](https://hadalmedusa.com/services/seo/)