Indexed, though blocked by robots.txt: what it means and what to do

Google’s Page indexing report states it simply: the page was indexed despite being blocked by the website’s robots.txt file. Google knows the address and has not seen the content, so the result it shows has no proper description.

A robots.txt rule controls fetching, not indexing. If another page, on your site or someone else’s, links to a blocked address, Google learns that the address exists. If it judges the address worth listing, it lists it, using the link’s text in place of a description.

The right fix depends on the answer to one question: should this page be in Google?

Remove or narrow the robots.txt rule that blocks it. Google can then read the page, and indexes it properly with its title and description. Our guide to blocked by robots.txt shows how to find the rule.

Blocking is the wrong tool, as this status shows. Do the opposite of what seems natural.

1. Remove the block, so that Google can fetch the page.
2. Add a noindex instruction to the page.
3. Wait for Google to fetch it and see the instruction. The page then leaves the index.

Keep the block in place and the page can stay listed for as long as anything links to it. Our guide to excluded by noindex tag covers the instruction.

A handful of basket, account or search addresses under this status, on a hosted platform that blocks them by default, do no harm. They hold no content worth ranking, and removing the platform’s rule to tidy the report can create more addresses to crawl than it removes.