"Indexed, though blocked by robots.txt" means Google has added a page to its index even though your robots.txt file told it not to crawl that page. This happens because robots.txt controls crawling, not indexing. Blocking a URL stops Google from reading the page, but it does not stop Google from listing the URL if it finds it another way. It also means Google can't see a "noindex" tag on that page, because it never crawls it.
So there are really two problems hiding behind this status, and they have opposite fixes:
- A page you want indexed is blocked by mistake. You need to unblock it.
- A page you don't want indexed is getting indexed anyway. robots.txt is the wrong tool for that, and this guide shows you the right one.
Crawling vs indexing: the one idea that explains everything
Crawling is Google reading your page. Indexing is Google storing it so it can show up in search. Your robots.txt file only affects the first one.
When you "Disallow" a URL in robots.txt, you tell Google not to fetch it. But if Google discovers that URL somewhere else, it can still index the address on its own, usually with no description because it never saw the content. That is the "Indexed, though blocked by robots.txt" state.
Why would Google index a URL it can't crawl?
Two things cause it:
- Links to the page. If other pages link to the URL (yours or someone else's), Google treats it as worth knowing about and can index the address, even without crawling it.
- Your sitemap. If the URL is in your XML sitemap, you are actively telling Google the page is meant to be found, which pushes toward indexing.
Where to check for this issue
In Google Search Console, open Indexing > Pages, then look for the reason "Indexed, though blocked by robots.txt." Click it to see the affected URLs, select one, and use Inspect URL to confirm the crawl was blocked.
You can also catch it without waiting for a Search Console notification. SiteGuru's indexation report shows this from both sides in one place (more on that below), so you can spot a page that's blocked but still indexed before it becomes a problem.
Fix 1: a page you want indexed is blocked by mistake
Step 1: check whether the block was intentional
First, decide if the page should be blocked at all. A robots.txt rule written by you or a developer is fine, as long as:
- You are not blocking pages that should rank for a keyword.
- You haven't set a broad rule that accidentally catches pages you want indexed.
If the block is intentional and you are happy for the page not to be crawled, you're done. If it isn't, keep going.
Step 2: find and fix the rule that's blocking it
Open your robots.txt file (always at the root, for example https://yourdomain.com/robots.txt) and check the directives. Watch for:
- A single
User-agentblock, not several conflicting ones. - A
Disallowline that doesn't sit directly underUser-agentwith no value in between. - Invisible characters. Paste the file through a plain-text editor to strip any stray encoding.
If you want search engines to crawl everything, your file should read like this. Note the empty Disallow value, which means "block nothing":
User-agent: *
Disallow:
Do not write Disallow: / here. That blocks your entire site, which is the opposite of what you want.
Find which line is blocking a specific URL. In Google Search Console, open the URL Inspection tool, paste the URL, and click Test Live URL. Google tells you inline whether robots.txt is blocking it. To review the file itself and any parse errors, use the robots.txt report under Settings > robots.txt. Once you know which Disallow line is responsible, remove or narrow it so the important pages are allowed.
Are you editing the robots.txt that's actually live?
You can only have one robots.txt per site, served from the root. The common trap is not "duplicate files," it's editing the wrong source of that one file.
WordPress serves a virtual robots.txt that it generates on the fly, and SEO plugins edit that virtual version. But if a physical robots.txt file exists in your web root, it overrides the virtual one. So you can edit the file in your plugin, see no change live, and be confused. If your edits aren't taking effect, check for a physical robots.txt file on the server first.
To edit the robots.txt your plugin controls:
- Yoast SEO: wp-admin > Yoast SEO > Tools > File Editor, then edit and save.
- Rank Math: wp-admin > Rank Math > General Settings > Edit robots.txt.
- All in One SEO: wp-admin > All in One SEO > Robots.txt.
Other things that can block the wrong pages
- Redirect chains. Too many hops waste crawl budget and confuse bots. Keep redirects short and mapped.
- Canonical tags. If you use a canonical tag to point at the "main" version of a page, make sure it isn't quietly pointing Google away from a page you want indexed.
- URL variations. Google may pick up a UTM parameter or an odd URL variant. Confirm whether it's a real page and tidy the URL if not.
Step 3: validate the fix
After editing, confirm everything is right:
- Re-test the URL with URL Inspection > Test Live URL and confirm it's no longer blocked.
- Check the robots.txt report for syntax errors.
When it looks good, go to Indexing > Pages, select the issue, and click Validate Fix so Google re-checks the affected URLs.
Fix 2: a page you don't want indexed keeps getting indexed
This is the case people get wrong most often. If a page is showing up in search and you want it gone, robots.txt is not the fix. Blocking crawling actually prevents Google from seeing the instruction that removes a page.
Use one of these instead:
- Add a
noindextag (<meta name="robots" content="noindex">or anX-Robots-Tagheader) and make sure the page is not blocked in robots.txt, so Google can crawl it and see the tag. Learn more in our noindex guide. (Note: aNoindex:line inside robots.txt does nothing. Google stopped supporting it in 2019.) - Password-protect the page or folder on your server. Anyone, including Google, can read your robots.txt, so it's not a way to hide private content.
- For a recent migration: if you no-indexed or removed old URLs, add 301 redirects from old to new (this preserves link equity), then give Google time to drop the old URLs.
How to find the pages you should be no-indexing
- List your URLs. Do it manually or with SiteGuru's crawler for full coverage.
- Pick the ones that shouldn't rank (staging, thin or duplicate pages, internal search results, account pages) and add a
noindextag to each. - Remove internal links pointing to pages you're de-indexing, so Google stops rediscovering them. GSC won't show you those links, but SiteGuru will.
- Re-audit to confirm the right pages are indexable and the rest carry a noindex.
See both sides at once with SiteGuru's indexation report
Google Search Console tells you the outcome: whether a page ended up indexed. It doesn't line that up against what your own site is telling Google to do. That gap is where "Indexed, though blocked" hides.
SiteGuru's indexation report shows both sides together, per page:
- Your signals: whether the page is blocked by robots.txt, carries a meta robots
noindex, and where its canonical points. - Google's reality: the actual indexation and coverage state Google reports for that page.
Putting them next to each other makes the mismatches obvious: a page you meant to block that's indexed anyway, or a page you want ranking that's quietly excluded. You get this on a weekly schedule, so you can act on the conflicts and skip the manual cross-referencing.
Editing robots.txt on Shopify
You used to be unable to edit the Shopify robots.txt file. Now you can. Go to Online Store > Themes > ... > Edit code > Add a new template, choose robots, and create robots.txt.liquid. From there you can add your rules and exceptions.
Keep an eye on your coverage
Seeing a mix of statuses in Search Console is normal. The skill is knowing when to act. For "Indexed, though blocked by robots.txt," keep your robots.txt clean and reserve noindex for pages you truly want out of search. Then monitor it, by hand or with SiteGuru's weekly audits, so you only dig into the technical detail when something actually needs you.
FAQ
Can I block crawling for my entire website?
Yes, with User-agent: * and Disallow: /. But remember that URLs can still be indexed if they're linked elsewhere, because the block stops crawling, not indexing. This rule also doesn't affect the AdsBot crawlers, which must be named explicitly, so you can block search bots and still run ads.
How do I block crawling of a single directory?
Follow the directory name with a slash:
User-agent: *
Disallow: /tags/
That blocks everything under /tags/. As always, use proper authentication for anything genuinely private, since robots.txt is public and blocked URLs can still be indexed without being crawled.
