AI Quick Rank Diagnostics

AI Crawler

An AI crawler or user agent fetches web pages for model training, search, or a request made by a person using an AI tool.

Also known as AI agent, LLM crawler. Updated .

What it means

The crawler name tells you who sent the request. Its purpose tells you what the visit may do for your site. Check both before you change robots.txt.

A training crawler collects material that may help build future AI models. A search crawler builds or refreshes an index used in AI answers. A user-requested agent opens a page because somebody asked an AI tool to use it right now.

The part a short definition leaves out

Those jobs do not share one universal robots rule. OpenAI says robots.txt may not apply to ChatGPT-User. Perplexity says Perplexity-User generally ignores it. Anthropic says ClaudeBot, Claude-SearchBot, and Claude-User honor its directives.

QuickRankAI treats a broad block as something to inspect because the right setting depends on the access you meant to allow. The useful question is not whether AI visited. It is which agent asked, why it came, and whether the page returned the content you expected.

Three AI crawler jobs need different controls

One company can use separate agents for separate jobs. A single allow or block may therefore change training, search visibility, or user-requested retrieval without changing the other two.

  1. Model training. GPTBot and ClaudeBot collect material that may help train future models. Both companies publish robots.txt controls for these agents.
  2. AI search. OAI-SearchBot, Claude-SearchBot, and PerplexityBot build or refresh search indexes. These agents use published robots.txt controls.
  3. User-requested retrieval. ChatGPT-User, Claude-User, and Perplexity-User open pages for a person using an AI tool. Claude-User honors robots.txt. OpenAI says it may not apply to ChatGPT-User. Perplexity says its user agent generally ignores it.

Check the exact AI agent before changing access

  1. Open the live robots.txt file. Read the group that matches the exact agent instead of assuming the wildcard group tells the whole story.
  2. Name the job. Decide whether the agent supports training, search, or a person asking for the page.
  3. Check the request. Match the user agent with the company's published IP ranges when those ranges are available.
  4. Read the response. Confirm the raw HTML contains the useful answer before JavaScript runs.
  5. Run a QuickRankAI inspection. Review the robots finding beside the raw HTML and page-structure checks so you fix the block without missing the page behind it.

A block can be deliberate. An inherited block can be expensive. The check is there to tell the difference before you change it.

Where this comes from

Related terms

  • robots.txt A plain text file at the site root telling crawlers which parts of the site they may request.
  • Client-Side Rendering Building page content in the browser with JavaScript after the server has already sent its response.
  • llms.txt A plain text file at the site root describing what a site covers and which pages matter most, written for AI clients.

See whether this is costing you anything

The inspection reports on this signal directly, alongside thirty-seven others, and tells you whether it is a quick fix on your site or something deeper.

Run a Quick Inspection Full glossary