AI + Search Exposure Inspection

AI Crawler

An automated client operated by an AI company that fetches pages for training data or live answer retrieval.

Also known as AI agent, LLM crawler. Updated .

What it means

GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended, CCBot and others. Some collect training data. Some fetch pages live to answer a question somebody just asked. They are not the same thing and they do not always respect the same rules, but they all read robots.txt and they all read the raw HTML response.

The part a short definition leaves out

Blocking these agents is a legitimate business decision, and plenty of publishers make it deliberately. QuickRankAI flags the block because most of the sites carrying one did not choose it. It arrived in a template, a plugin default, or a security recommendation someone applied without reading.

Where this comes from

Related terms

  • robots.txt A plain text file at the site root telling crawlers which parts of the site they may request.
  • Client-Side Rendering Building page content in the browser with JavaScript after the server has already sent its response.
  • llms.txt A plain text file at the site root describing what a site covers and which pages matter most, written for AI clients.

See whether this is costing you anything

The inspection reports on this signal directly, alongside thirty-seven others, and tells you whether it is a quick fix on your site or something deeper.

Run a Quick Inspection Full glossary