WordPress Noindex: How to Avoid Blocking Your Website from Google and AI

TL;DR Noindex is a page-level rule delivered through the meta robots tag or an X-Robots-Tag header, not through robots.txt. The biggest trap: a robots.txt Disallow cancels noindex, because Google never fetches the page and never sees the tag. Noindex speaks to search engines; AI bots like GPTBot, ClaudeBot and PerplexityBot read robots.txt, so the two systems need separate checks. Quick path: check the WordPress Reading setting, then your SEO plugin, then robots.txt, then verify in Search Console and in an AI visibility tool. Semly shows whether your key pages still appear as sources in AI answers, so you can catch the problem before it costs you visibility.

Your WordPress site can look perfectly healthy and still be invisible. No error, no broken layout, no warning in the dashboard. Just silence in Google and in AI answers.

The usual suspect is a noindex rule you did not mean to publish. It often comes from a single checkbox, a plugin default, or a leftover setting from a staging site. The good news: it is fixable, and the fix takes minutes once you know where to look.

This guide walks through what noindex really does, where WordPress hides it, how to diagnose it step by step, and how to confirm the fix in both Google and AI search.

What WordPress noindex actually does (and what it doesn't)

Noindex is a rule that tells search engines: do not put this page in your index. It arrives in one of two ways:

• A meta robots tag in the page HTML: <meta name="robots" content="noindex">
• An HTTP header: X-Robots-Tag: noindex

That is it. Noindex is a page-level instruction. It is not a crawl rule, and it is not a file on your server.

Here is the sentence worth remembering: noindex works only if the page is not blocked in robots.txt. If a crawler is not allowed to fetch the page, it never reads the tag. Google states this directly, and it also confirms that writing noindex inside robots.txt is not supported at all. A Disallow line simply voids your noindex.

A few related terms get mixed up constantly:

• nofollow is about links, not about the page itself. It does not remove a page from the index.
• robots.txt Disallow manages crawling. It is not a deindexing tool. A blocked URL can still show up in results, just without a description.
• canonical is a consolidation hint. It tells Google which URL you prefer, but it does not block anything.

So: does noindex remove a page from Google? Yes, once Google crawls the page and sees the tag. Is nofollow the same thing? No. Will robots.txt remove a page from results? No, and it can even make things worse.

How WordPress emits noindex (settings, plugins, and the traps)

WordPress can generate noindex from several places. Check all of them, because one is enough to hide your site.

Settings → Reading → Search engine visibility. Since WordPress 5.3 (November 2019), ticking "Discourage search engines from indexing this site" outputs noindex,nofollow across the site. The value is stored in the wp_options table as blog_public, where 0 means discouraged and 1 means public. In a multisite network, this setting is per-site, so one subsite can be hidden while the rest are fine.

Yoast SEO. Per post, open the meta box → Advanced tab → "Allow search engines to show this post in search results?" set to No. Yoast defaults to noindex, follow, so links are still followed. Internal search pages are noindexed by default and cannot be changed from the interface.

Rank Math. Switch to Advanced Mode, then go to Titles & Meta → Robots Meta and set No Index for a post type or taxonomy. Per-post control lives in the Advanced tab. The PRO version adds Quick Edit and bulk actions.

All in One SEO. Offers both a global No Index toggle and per-post robots settings.

What the WordPress setting does not do: it does not add X-Robots-Tag headers to images, PDFs, or video files. WordPress core cannot set that header for directly served assets, so this is a job for your server or host.

Watch out: staging to production The blog_public = 0 value travels inside your database. When you push a staging site live, the checkbox often comes along with it. Hosting tools may check the box automatically when going live to staging, but they do not always uncheck it on the way back. Always verify this setting right after a migration.

Step-by-step: diagnose and fix noindex on your WordPress site

Work through these in order. Each step has a clear pass or fail.

1. Check the WordPress setting. Go to Settings → Reading and make sure the visibility checkbox is unchecked. Confirm the database value with WP-CLI: wp option get blog_public. Pass: the value is 1.
2. Check the page source and headers. Run curl -sI https://yourdomain.com | grep -i x-robots-tag and curl -s https://yourdomain.com | grep -i 'name="robots"'. Pass: no noindex in either result.
3. Check your SEO plugin. Review global settings for each post type and taxonomy, then spot-check individual pages. Pass: no global No Index is active.
4. Check robots.txt. Fetch https://yourdomain.com/robots.txt and look for Disallow: / plus any AI bot directives. Pass: no site-wide Disallow.
5. Check Search Console. Open URL Inspection. If you see "Indexing allowed? No: 'noindex' detected", the tag is live. Pass: "Indexing allowed: Yes".
6. Request indexing. In URL Inspection, click Request Indexing to nudge a re-crawl.
7. Check AI visibility. Confirm whether your key pages still appear as sources in ChatGPT, Gemini, Perplexity, and Claude.

One expectation to set: Google has to revisit the page before anything changes. That can take days, and for less important pages it can stretch into months. The temporary removal tool in Search Console hides a URL for roughly six months, which is an approximation, not a permanent fix.

If the tag is still there after all of this, go back to step 3. A plugin or theme filter can override your settings, and a global No Index for a post type is easy to miss.

The robots.txt trap: when Disallow cancels your noindex

This is the mistake that confuses the most people, so it deserves its own section.

Disallow blocks crawling. If Googlebot cannot fetch the page, it cannot read the meta tag. Your noindex becomes a message nobody receives. The URL can stay in the index, sometimes without a description, because Google may still learn about it from links pointing to it.

The result is the worst of both worlds: the page is blocked from crawling and still visible in search results.

Google's rule is simple: the more restrictive rule wins. When several crawlers are named with different rules, Google applies the sum of the negative rules.

Decision tree: what do you actually want?

• Want the page gone from the index? Use noindex. Do not use Disallow.
• Want to reduce crawl traffic on a section? Use Disallow, and accept that the page may still be indexed.
• Want both gone and blocked? Remove the Disallow first, let noindex do its job, then reconsider crawl rules later.

If you already have both in place, delete the Disallow line, keep the noindex, and request indexing.

Don't block the AI bots you actually want (training vs search vs user)

Here is the part most WordPress guides skip. Noindex is a search engine rule. AI bots are governed by robots.txt tokens, not by meta robots. A site-wide noindex does not stop GPTBot from crawling, and a robots.txt block does not remove you from Google.

AI crawlers now come in three flavors:

• Training: GPTBot, ClaudeBot, Google-Extended
• Search and retrieval: OAI-SearchBot, Claude-SearchBot, PerplexityBot
• User-triggered: ChatGPT-User, Perplexity-User, Claude-User

Two things matter here. First, blocking search bots removes your brand from AI answer citations. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, and Anthropic notes that blocking Claude-SearchBot may reduce visibility in search results. Second, Google-Extended does not affect your inclusion in Google Search and is not a ranking signal, so blocking it is a training decision, not a visibility one.

The stakes are growing. Cloudflare's crawler research (May 2024 to May 2025, cited research) recorded GPTBot requests up 305% and PerplexityBot requests up 157,490% year over year. A wrong token in robots.txt now has a much bigger effect than it did a year ago.

Robots.txt changes typically propagate within about 24 hours.

Verify your AI visibility after the fix

Search Console only tells you about Google. It says nothing about whether ChatGPT, Gemini, Perplexity, or Claude still cite your pages.

That is a separate check, and it is worth doing properly. Ask the models questions your customers would ask, then look at which sources they reference. If your key pages have dropped out, you have a visibility gap even if Google looks fine.

Semly monitors this daily. The platform analyzes prompts, sources, competitors, and generated responses to show whether your brand, your competitors, or your products are being recommended by AI systems. It tracks Share of Model Voice, sentiment, and content gaps, so you can see where you are missing from answers and what to publish next. You can start with a free AI visibility report.

Monthly checklist:

• Confirm key pages still appear as sources in AI answers
• Re-check robots.txt for accidental AI bot blocks
• Review Search Console for new noindex detections
• Compare your Share of Model Voice against competitors
• Re-run the WordPress Reading setting check after any migration

Common mistakes and how to fix them

1. The discourage checkbox left on after migration. Symptom: the whole site disappears from Google. Cause: blog_public = 0 carried over in the database. Fix: uncheck it, verify blog_public = 1, then request indexing.

2. Disallow: / combined with noindex. Symptom: the page stays in results without a description, and noindex seems to do nothing. Cause: robots.txt blocks the crawl, so the tag is never read. Fix: remove the Disallow line and keep the noindex.

3. A global No Index for a post type. Symptom: every post in one section vanishes. Cause: a plugin setting applied at the post type level. Fix: correct it in the SEO plugin's Titles & Meta settings.

4. Blocking search bots instead of only training bots. Symptom: your brand stops appearing in AI answers. Cause: OAI-SearchBot, Claude-SearchBot, or PerplexityBot blocked in robots.txt. Fix: restore those tokens and keep training decisions separate.

5. No X-Robots-Tag on files. Symptom: images and PDFs stay indexed on a site you thought was hidden. Cause: WordPress cannot set headers for directly served assets. Fix: configure the header at the server or host level.

6. Multisite blind spots. Symptom: one subsite is invisible while the rest are fine. Cause: blog_public is per-site. Fix: check every site in the network individually.

If you do not have server access, ask your host to add the headers for you. That is a normal request and most hosts handle it quickly.

What to do next

Right now: uncheck the visibility setting, fix any Disallow: / line, and confirm blog_public = 1.

This week: run URL Inspection in Search Console, request indexing for your most important pages, and audit robots.txt for AI bot tokens.

Every month: re-check robots.txt and AI bot directives, review Search Console for new noindex detections, and monitor your AI visibility so a quiet drop does not go unnoticed.

Pick the fix that matches your situation:

• A development or staging site: use an X-Robots-Tag header at the server level instead of the WordPress checkbox.
• A few individual pages: use per-post noindex in your SEO plugin.
• The whole site: use the global setting, then verify it after every migration.

When you lack server access, ask your hosting provider for the headers. When you want to know whether the fix actually restored your presence in AI answers, Semly makes that a standing part of your monitoring rather than a one-time check.

FAQ

Does noindex remove a page from Google? Yes, once Google crawls the page and reads the tag. The page is dropped from results even if other sites link to it. The catch is that Google must re-crawl the page first, which can take days to months.

Does noindex block AI bots? No. Noindex is a search engine directive. AI bots like GPTBot, ClaudeBot, and PerplexityBot follow robots.txt tokens, so you need to manage those separately.

How long does deindexing take? It depends on how often Google revisits the page. Important pages get recrawled faster. Requesting indexing in Search Console can speed things up, but there is no guaranteed timeline.

Will Disallow remove a page from search results? No. robots.txt manages crawling, not indexing. A disallowed URL can still appear in results, sometimes without a description, and it also cancels any noindex you set on that page.