Your robots.txt Is Blocking the Crawlers You Want

The AI crawler block is the most common finding QuickRankAI reports, and almost nobody who has one chose it. It takes one line to remove.

By Krisada. Updated September 7, 2026.

If your robots.txt blocks GPTBot, ClaudeBot, PerplexityBot, or Google-Extended, you are not in AI answers, regardless of how good the pages are. It is the most common finding this inspection reports, and in almost every case the site owner did not decide it. A plugin default, a security recommendation, or a copied file decided it.

The block usually looks like this

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

Sometimes it is broader and worse:

User-agent: *
Disallow: /

That second one blocks everything, and it appears more often than you would think, usually because it was correct on staging and travelled with the deploy.

Which agents are which

The distinction between training and live retrieval is worth holding onto, because it is where the real decision sits. Blocking training crawlers is a defensible position about how your work gets used. Blocking live retrieval agents means you are not in the answer when somebody asks a question you spent a year answering.

Plenty of publishers block the training crawlers deliberately and let the retrieval ones through. That is a coherent position. What is not coherent is blocking both because a plugin shipped with it on.

The rule that trips people up

Longest matching rule wins between Allow and Disallow. So this:

User-agent: *
Disallow: /blog
Allow: /blog/important-post/

allows that one post and blocks everything else under /blog. And this:

Disallow: /

with no Allow beneath it blocks the entire site for that group, including the pages you are inspecting.

Also note that a Disallow line with no value at all means the opposite of what it looks like. It grants everything.

Blocking is not removing

A blocked URL can still show up in results, because the crawler never fetched it and therefore never saw whatever directive you also put on the page. If you want a page out, let the crawler fetch it and use noindex. If you want to save crawl budget on something worthless, block it. Two different tools.

Declare your sitemap while you are in there

Sitemap: https://example.com/sitemap.xml

One line, and it is the only way a crawler that has never seen your site learns where the list of your URLs lives. QuickRankAI checks for this separately, and a surprising number of sites have a sitemap that nothing knows about.

How to confirm the repair landed

curl -s https://example.com/robots.txt

Read the whole file, not just the top. The rule that is causing the problem is frequently at the bottom, added later by something else. Then re-run the inspection, which names each blocked agent individually rather than just reporting that something is blocked.