How to Check Whether AI Crawlers Can Access Your WordPress Website (and Fix It If They Can't)

TL;DR AI crawlers have three jobs (training, AI-search indexing, user-triggered fetch) and each one uses its own token, so "block OpenAI" is really several separate decisions. robots.txt is a declaration, not a security layer, and user-triggered fetchers may ignore it. On WordPress the usual culprits are security plugins, CDN bot-fight modes and hosting-level bot mitigation, not robots.txt itself. You diagnose with robots.txt, HTTP status codes and server logs. You fix it with allowlists in robots.txt, your WAF/CDN and your server config. After the change, wait about 24 hours and test again.

Why AI crawlers WordPress access is the first thing to check

If your WordPress site never shows up in ChatGPT, Perplexity or Claude answers, the problem is often not your content. It is access. A crawler that gets a 403 never reads your page, so nothing else you do matters yet.

AI crawlers come in three roles, and each role is a separate decision:

• Training crawlers collect content that may feed model training. Blocking them keeps your text out of training sets.
• AI-search indexing crawlers build the index that lets assistants cite and link you. Blocking them removes you from AI answers.
• User-triggered fetchers grab a single page because a real person asked a question about it. These are the ones that may ignore robots.txt.

That is why "block OpenAI" is not one switch. OpenAI alone runs several tokens, and Anthropic and Perplexity do the same.

Watch out: Google-Extended and Applebot-Extended are control tokens, not crawlers. They never send a request, so they will never appear in your logs. Seeing zero hits for them is normal, not proof that a block works.

And remember what robots.txt really is. Under RFC 9309 it is a voluntary declaration, not access control. Listing a path there can even make it more discoverable. If you need real protection, use authentication or network rules.

What you need before you start

Get these in place first, or your diagnosis will stall halfway.

• Access to your robots.txt file (FTP, hosting panel, or your SEO plugin).
• Access to your CDN dashboard, for example Cloudflare.
• Access to server logs, or a logging plugin if your host does not expose raw logs.
• A terminal with curl, or an online header-checking tool.
• Access to Google Search Console.

When this method will not work: if you are on fully managed hosting with no access to logs, CDN settings or server config, you cannot complete steps 3 and 4 yourself. In that case, send your host a support ticket asking them to allow the AI crawler tokens listed above. You can still do steps 1, 2 and 5.

Step 1: Read your robots.txt and spot the blocks

Open https://yourdomain.com/robots.txt in a browser. You want a plain text file, not an HTML error page. It should be UTF-8 and served as text/plain.

Then check how the rules match:

• User-agent matching is case-insensitive.
• If several groups match the same token, they get merged.
• If no specific group matches, the * group applies.
• The longest, most specific match wins.
• If an allow and a disallow rule are equally specific, allow wins.

A clean allow block for the citation crawlers looks like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Applebot
Allow: /

Do not block Googlebot. It handles classic Google indexing and also feeds AI Overviews. Blocking it to "keep AI out" costs you normal search traffic.

Watch out: SEO plugins like Yoast, Rank Math and All in One SEO can generate a virtual robots.txt that overrides or appends to your physical file. If your edits keep disappearing, the plugin is the source of truth. Check its settings, not just the file.

Checkpoints:

• The file returns 200.
• There is no Disallow: / for *.
• No AI search tokens sit in a disallow section if you want to be cited.

Step 2: Test access with curl and check HTTP status codes

robots.txt only tells you what you declared. curl tells you what the bot actually receives.

curl -A "GPTBot" -I https://yourdomain.com/
curl -A "OAI-SearchBot" -s -o /dev/null -w "%{http_code}" https://yourdomain.com/

Then read the code:

A 403, 429 or 503 is the classic silent block. It never shows up in robots.txt, which is exactly why so many site owners miss it.

Checkpoints: repeat the test for OAI-SearchBot, ClaudeBot and PerplexityBot. The telltale symptom is a 200 in your browser but a 403 for the bot.

Step 3: Check CDN, WAF and hosting-level blocks

This is where most WordPress blocks actually live.

Cloudflare: go to Security, then Bots. Cloudflare classifies AI bot behaviour into Search, Agent and Training, so you can allow by behaviour instead of one blunt switch. Check the "AI Scrapers and Crawlers" toggle. If it is on, AI bots are blocked. Note that Cloudflare's newer defaults block Training and Agent bots on ad-displaying pages while Search stays allowed, and mixed-purpose crawlers get caught by any training-block configuration.

To allow a specific bot, add a WAF custom rule with the action Allow, and give it a higher priority than your blocking rules.

Wordfence and similar plugins: their firewalls and rate limiting can 403 AI bots before robots.txt is ever read. Add the crawler user-agents to the allowlist.

Managed hosting: many hosts apply server-level bot mitigation that silently blocks crawlers. Ask support to confirm.

The rule that matters: allowlist by User-Agent and IP range. A user-agent string alone is easy to spoof. OpenAI and Perplexity publish their ranges as JSON files, so pull those and refresh them periodically.

Checkpoints: the allow rule outranks the block rule, you re-test after the change, and you did not accidentally block Googlebot along the way.

Step 4: Verify with logs and Search Console

Now confirm the bots actually arrive.

• Filter your server logs for GPTBot, ClaudeBot and PerplexityBot.
• Verify identity with reverse DNS or the operator's published IP files, since user-agents can be faked.
• In Google Search Console, use the robots.txt report to test a URL against a specific user-agent, and URL Inspection for a live check.

Watch out: robots.txt changes can take about 24 hours to propagate. OpenAI and Perplexity both document that lag. And again, no Google-Extended or Applebot-Extended entries in your logs is expected.

Checkpoints: re-test after roughly 24 hours. If you still see 403, go back to step 3.

Step 5: Apply the fix and re-test

Work in this order, one layer at a time:

1. robots.txt
2. WAF and CDN rules
3. Server configuration (nginx map blocks, Apache .htaccess)

Only after that should you touch content and llms.txt. Think of llms.txt as a curated map of your key resources, not a replacement for robots.txt or your sitemap. Current evidence does not show a measurable citation impact from it yet, so treat it as a helpful addition, not a fix.

A ready block to paste:

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Applebot
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

Do not remove existing rules without reading them first. And do not ship several changes at once, or you will not know which one broke something.

Checkpoints: run curl again, re-test in Search Console, and review the logs once more.

Common mistakes and how to fix them

• Blocking * instead of specific tokens. Symptom: everything disappears from search. Fix: replace the wildcard block with named tokens.
• Writing the version string, like GPTBot/1.4. Symptom: the rule never matches. Fix: use the substring GPTBot only, since versions change without notice.
• Blocking Googlebot together with AI bots. Symptom: lost organic traffic. Fix: remove Googlebot from the disallow list.
• Relying on robots.txt while your WAF blocks anyway. Symptom: bot still gets 403. Fix: add an allow rule at the CDN or firewall layer.
• Allowlisting by user-agent alone. Symptom: spoofed traffic gets through while real bots still fail. Fix: combine user-agent with the published IP ranges.
• Skipping the re-test. Symptom: you assume it works. Fix: re-check after about 24 hours.
• Expecting instant results. Symptom: nothing changes in an hour. Fix: allow for propagation and model update cycles.

What to do next

Once access is open, you have a baseline to build on. Run a free AI visibility report at report.semly.ai to see how ChatGPT, Gemini, Perplexity and Claude currently talk about your brand. Semly's recommendations engine flags exactly this class of problem, including blocked AI bots and robots.txt issues, and its Leon agent checks your robots.txt and llms.txt for you. If you publish with WordPress or WooCommerce, Semly's integrations let you push and update content without leaving your workflow.

Then keep the loop running: monitor logs, refresh llms.txt, review your structured data and keep content fresh.

Decision tree:

• Want to be cited in AI answers? Allow the search crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot.
• Want your content out of training sets? Block only the training crawlers: GPTBot, ClaudeBot, and similar.
• No access to the network layer? Ask your host or CDN support to allow the tokens for you.

Access is the foundation. Get it right, verify it, and everything else you do for AI visibility has a chance to work.