Blocked by robots.txt: what it means and how to fix it

The robots.txt file lists parts of a site that crawlers are asked not to fetch. Google’s Page indexing report puts a page under this status when a rule in that file matched its address. Google did not read the page, so it knows the address and nothing about the content.

Blocking is right for parts of a site with no value in search and many addresses: baskets, checkouts, account areas, internal search results. Hosted platforms block these by default. A list of such addresses needs no action.

The status needs attention when it lists pages that should be found. Three causes are common.

• A rule left from when the site was being built, often Disallow: /, which blocks everything.
• A rule written for one folder that also matches others, because rules match the start of an address.
• A plugin or app that added rules of its own.

Open your domain followed by /robots.txt in a browser. Read each Disallow line under User-agent: * and under any group naming Googlebot. The rule responsible is the one whose path begins the same way as the blocked address. Search Console’s robots.txt report shows the file as Google last fetched it.

Remove the rule, or narrow it so that it covers only what should be blocked. Where the file is edited depends on the platform: directly on the server, through an SEO plugin on WordPress, or in the platform’s settings on Wix and Shopify. Then request indexing for the pages that matter.

A blocked page can still turn up in search. Google’s guidance says so plainly: to make sure a page is not indexed, remove the robots.txt block and use a noindex instruction instead. A block stops Google reading the page, and so stops it seeing the instruction that would keep the page out. The companion status is covered in excluded by noindex tag, and the choices about AI crawlers in should you block AI crawlers?