Do AI crawlers obey robots.txt?

The major named crawlers document that they do. Unidentified scrapers may not, which is a separate problem that robots.txt cannot solve.

Guide · Powered by the full LLMao scan

AI Crawlers: Who They Are and What They Read

A plain reference to the crawlers behind AI search, what they respect, and how to verify your own configuration.

Three different jobs

Training and index crawlers gather content in bulk, ahead of time.
Search crawlers build the index behind an assistant's retrieval step.
Live fetchers request a single page when a user's question needs it.

Controls at your disposal

robots.txt rules per user agent.
Meta robots and X-Robots-Tag for page-level indexing.
Server-side rendering so fetched HTML actually contains the content.
An optional llms.txt summary describing your site's key pages.

What LLMao checks for ai crawler accessibility

These are real tests from the LLMao methodology — the same ones that run in every scan, covering Technical Accessibility, Content Freshness.

AI Crawler Access

Technical Accessibility

robots.txt allows GPTBot, ClaudeBot, PerplexityBot, Google-Extended

JavaScript Dependency

Technical Accessibility

Content accessible without excessive JavaScript rendering

Meta Description

Technical Accessibility

Present, descriptive, and within character limits

Social & OG Metadata

Technical Accessibility

Open Graph and Twitter Card metadata present

The same scan also examines content structure, readability, schema.org markup, entity definition, authority & trust signals, citation & source quality — you always get the complete report. See the full methodology.

Frequently asked questions

Keep reading

AI crawler checkerCheck whether your website is reachable by AI and search crawlers as part of a complete AI visibility analysis.robots.txt checkerSee how your crawl configuration reads to AI assistants — checked inside the full LLMao analysis, not by a separate one-off script.ChatGPT SEOChatGPT reaches the web in two ways — a crawler that gathers content and a browsing tool that fetches pages live. Both need to succeed on your site.llms.txt Explained: What It Is, Who Reads It, and How to Write Onellms.txt is a small markdown file at your site root that tells AI assistants what your site is and where the important pages live. Here's what it does, what it doesn't do, and a template you can ship today.

All articles Methodology How it works

One scan. Your complete AI readiness report.

Crawler access, content structure, structured data, entity clarity, metadata, trust signals and more — 8 categories, 35+ tests, one report you can save and rescan.

Run a free scan