LLMaoLLMao

    Free check · Powered by the full LLMao scan

    AI Crawler Checker

    Check whether your website is reachable by AI and search crawlers as part of a complete AI visibility analysis.

    Free single-page analysisNo credit card requiredResults in 30 seconds

    No URL handy? See a sample report

    This runs the complete LLMao scan — the same engine, methodology and scoring used everywhere on the site. You get the full report; we simply open it on the part you came here for.

    Why crawler access decides everything else

    AI assistants can only summarise, quote or recommend pages they are allowed to fetch. If your robots.txt blocks GPTBot, ClaudeBot, PerplexityBot or Google-Extended, everything else on the page — great content, perfect schema, strong authority — is invisible to those systems.

    Blocking is often accidental: a staging rule that shipped to production, a CDN default, or a plugin that added a blanket disallow for unknown user agents.

    The main AI crawlers worth knowing

    • GPTBot — OpenAI's crawler used to fetch and index public web content.
    • OAI-SearchBot and ChatGPT-User — used when ChatGPT browses the live web.
    • ClaudeBot — Anthropic's crawler.
    • PerplexityBot — Perplexity's index crawler for cited answers.
    • Google-Extended — controls use of your content by Google's generative features.
    • Bingbot — still the backbone of several AI answer experiences.

    Server-rendered content matters too

    Most AI crawlers do not execute JavaScript the way a browser does. A page whose main text only appears after client-side rendering can be technically reachable and still return almost nothing useful. The scan flags heavy JavaScript dependency alongside crawler rules.

    What LLMao checks for ai crawler accessibility

    These are real tests from the LLMao methodology — the same ones that run in every scan, covering Technical Accessibility, Content Freshness.

    AI Crawler Access

    Technical Accessibility

    robots.txt allows GPTBot, ClaudeBot, PerplexityBot, Google-Extended

    JavaScript Dependency

    Technical Accessibility

    Content accessible without excessive JavaScript rendering

    Meta Description

    Technical Accessibility

    Present, descriptive, and within character limits

    Social & OG Metadata

    Technical Accessibility

    Open Graph and Twitter Card metadata present

    The same scan also examines content structure, readability, schema.org markup, entity definition, authority & trust signals, citation & source quality — you always get the complete report. See the full methodology.

    Frequently asked questions

    Keep reading

    robots.txt checkerSee how your crawl configuration reads to AI assistants — checked inside the full LLMao analysis, not by a separate one-off script.AI crawlersA plain reference to the crawlers behind AI search, what they respect, and how to verify your own configuration.Schema checkerCheck whether your website exposes structured data that AI assistants can actually use — as part of a complete AI visibility analysis.llms.txt Explained: What It Is, Who Reads It, and How to Write Onellms.txt is a small markdown file at your site root that tells AI assistants what your site is and where the important pages live. Here's what it does, what it doesn't do, and a template you can ship today.How ChatGPT, Claude & Perplexity Choose Which Websites to CiteAn in-depth look at the factors AI assistants use to determine which sources to reference and recommend to users.
    All articlesMethodologyHow it works

    One scan. Your complete AI readiness report.

    Crawler access, content structure, structured data, entity clarity, metadata, trust signals and more — 8 categories, 35+ tests, one report you can save and rescan.

    Run a free scan