About the AI Crawler View

Most SEO tools render your page like a browser. That is useful for Google, which executes JavaScript, but it is misleading for AI search, which largely does not.

We do the opposite. We make two requests to the same URL: one identifying as GPTBot, one as a normal browser. Comparing the two answers a question no rendered crawl can: is the problem your content, or is it your server?

You get one of these verdicts:

• AI crawlers can read this page. Your content is in the HTML. Crawlers can read and cite it.

• AI crawlers get only part of this page. Some of what a visitor sees is missing or restricted. The listed fixes close the gap.

• This page is effectively invisible to AI. Your content is rendered by JavaScript. Crawlers receive a near-empty shell, so there is nothing for them to quote.

• Your server blocks AI crawlers. A browser loads the page normally, but a request identifying as an AI crawler is refused before it ever reaches your content. This is a firewall, CDN or bot-protection rule, not robots.txt, and no amount of on-page work will fix it.

• Your site refuses automated requests. Neither agent could reach the page. We deliberately do not claim AI crawlers are being singled out here, because from a single vantage point that is genuinely unknowable.

Alongside the verdict you get the crawler's actual view: title, meta description, H1, every heading it found, structured data types, the framework detected, and a preview of the readable text with a word count. Plus an access check for eight AI crawlers and a look for an llms.txt file.

What it checks

The AI crawler's view of your page

Title, meta description, H1, all H2 and H3 headings, structured data types, canonical, robots meta, detected framework, HTML size, and the readable text with its word count.

AI crawler access, per bot

GPTBot (ChatGPT training), OAI-SearchBot (ChatGPT search), ChatGPT-User (ChatGPT browsing), ClaudeBot (Claude training), Claude-User (Claude browsing), PerplexityBot, Google-Extended (Gemini and AI Overviews) and CCBot (Common Crawl, which feeds many models).

Server-level blocking

Whether your infrastructure refuses AI user agents outright, which robots.txt checkers cannot detect.

llms.txt

Whether you publish the emerging convention for telling AI systems what your site covers.

Each issue comes with its severity and a specific fix, and you can copy the whole crawler view to share with a developer.

Use cases

Before you blame your content

If a page is not being cited, check that it is readable at all. Rewriting copy that a crawler never receives is wasted work.

After a site rebuild

Framer, Next.js, React, Vue, Nuxt and similar frameworks vary enormously in how much they render server-side. A redesign can silently move your content out of the HTML.

After a security change

Bot-protection rules and WAF settings are routinely tightened without anyone considering AI crawlers. This is the fastest way to catch it.

Auditing a new client site

A minute per key page tells you whether an AI search programme is even viable yet, and gives you a concrete technical finding for the pitch.

Settling the "should we block AI crawlers" question

See exactly which bots you currently allow, then decide deliberately rather than by inherited default.

Checking a competitor

The tool works on any public URL, so you can see whether rivals are readable, blocked, or publishing an llms.txt.

How to use it

1. Paste the full URL of the page you want to check, including https://.

2. Select Check what AI sees.

3. Read the verdict at the top. It tells you whether the page is readable, partly readable, invisible, or blocked.

4. Open What the crawler received and compare it with your live page. Anything missing here is missing from AI answers.

5. Check AI crawler access for any red dots, which show crawlers your robots.txt disallows.

6. Work through What to fix, which is ordered by severity, with high-severity items first.

7. Use Copy the AI view to send the findings to whoever maintains the site.

Check your highest-value pages first: your home page, core service or product pages, and any page you want quoted in an AI answer.

FAQs

Why does this show something different to my browser?

Your browser runs JavaScript, so it builds the finished page. AI crawlers largely do not. We show you the raw HTML response, which is what they actually work from.

Does it run JavaScript?

No, deliberately. Running it would hide the exact problem this tool exists to find.

My word count looks lower than what is on the page. Is that a bug?

Usually not. It is the point. If your visible copy is injected by JavaScript after load, it is not in the HTML, so a crawler never sees it. Compare the text preview against your live page to see what is missing.

What does "your server blocks AI crawlers" actually mean?

A normal browser request succeeds but a request identifying as GPTBot is refused. That is a firewall, CDN or bot-protection rule rejecting the user agent before your content is reached. Robots.txt checkers will not detect this, because the block sits in front of robots.txt.

How do I fix a server-level block?

Allow the AI crawler user agents in your WAF or bot-protection settings. On Cloudflare that is Security then Bots, or a WAF skip rule. Then re-run the check.

Should I be allowing AI crawlers at all?

That is a commercial decision, not a technical one. If you want to appear in AI answers, they need access. If you would rather not be used for model training, you can allow the search and browsing bots while disallowing the training ones. The tool separates them so you can tell which is which.

Does this tell me whether ChatGPT will cite me?

No, and be wary of anything that claims otherwise. This checks whether your page is reachable and readable, which is the prerequisite. Whether a model chooses to cite you depends on authority, relevance and competition. We would rather be accurate about the limit than sell you a number we cannot stand behind.

What is llms.txt?

An emerging convention, similar in spirit to robots.txt, for telling AI systems what your site covers and pointing them at your key pages. Adoption is early and support is not guaranteed, so we report it as a low-severity item rather than a must-have.

Does it check my whole site?

No, one page per check. Run it on the pages that matter most. If you need site-wide analysis, that is what our other tools and our audits are for.

Is it really free?

Yes, and there is no signup. We only fetch the public page, the same as any crawler would.

Want to be the answer, not a missing source?

Being readable is the floor, not the finish line. We help brands get found, read and cited by ChatGPT, Perplexity and Google AI, starting with making sure the machines can read the page at all. Questions about your results: [email protected]