Why AI assistants cannot read some websites

A site can look finished in a browser and still be empty, or closed, to the programs that fetch pages for AI assistants. There are two common causes. You can test for both in a few minutes.

Published
Reading time
6 minutes
Written by
Andy Biro, founder of Sharpfont

The short answer

  • An AI assistant can only use a page that its crawler was able to fetch and read.
  • Most AI crawlers do not run JavaScript. A page that is built in the browser reaches them with little or no text.
  • Hosts and firewalls can turn crawlers away before your site is reached, whatever your robots.txt says.
  • Neither problem shows in a normal browser. You have to look at the page the way a crawler does.

An assistant can only repeat what it could read

ChatGPT, Claude, Perplexity and Gemini do not look at a website the way you do. A program fetches the page for them. That program is called a crawler.

A crawler asks your server for an address and receives the page code that the server sends back. It reads the text in that code. If the text is not there, or the server refuses the request, the assistant has nothing from your site to work with. It then answers from other places: old profiles, directories, other people’s articles.

Two things most often stand between a good site and a crawler. Neither is visible in a browser.

Cause one: the page is built in the browser

Many modern sites are JavaScript applications. The server sends an almost empty page and a script. The script runs in the visitor’s browser and puts the text on the screen.

A person never notices. A crawler that does not run the script receives only this:

WHAT THE SERVER SENDS
<body>
  <div id="root"></div>
  <script src="/main.8f2a.js"></script>
</body>

There are no words in that page to read.

In December 2024 the hosting company Vercel and the search consultancy MERJ published a study of crawler traffic on Vercel’s network. They found that none of the major AI crawlers runs JavaScript. That includes the crawlers of OpenAI, Anthropic and Perplexity. Some of them download the script files, but they do not run them. Google is the exception: its crawler runs JavaScript, and Google’s own documentation describes how.

We found no newer measurement that says this has changed.

So a site built this way can be read by Google and still be blank to the crawlers behind ChatGPT, Claude and Perplexity.

Cause two: the server turns the crawler away

The second cause sits in front of your site. Hosting companies and security services run firewalls that decide which visitors get through. Some treat AI crawlers as unwanted bots. The crawler receives an error, or a “checking your browser” screen it cannot pass, in place of your page.

This is not the same as robots.txt. robots.txt is a text file on your site that asks crawlers to stay out of some or all of it. A firewall does not ask. It refuses. Your robots.txt can welcome every crawler while the firewall sends them away, and nothing in your own files shows it.

Blocking is becoming a default. On 1 July 2025 Cloudflare, a network that many websites sit behind, announced that it would block AI crawlers by default. Every new domain that signs up is now asked whether it wants to allow them.

A block can be exactly what a site owner wants. The problem is a block that nobody chose.

What happened to our own site

We found the second cause on our own site on 17 November 2026.

The site is static. Every word is in the page code, and our own check reported the home page as readable. Then we asked two assistants to open the site and say what the studio sells and what it costs.

  • Gemini read the site. It described the services and repeated every published price correctly.
  • ChatGPT could not open it. It reported a verification screen for automated visitors and built its answer from older profiles on other sites. That answer left out our current packages and quoted an hourly rate from an old freelance profile.

The cause was the hosting company’s firewall. It showed some automated visitors a “One moment, please...” screen in place of the page. Our robots.txt allowed every crawler. When we asked for the page under the name GPTBot, the server answered with a refusal.

We moved the site to a host that does not challenge crawlers. The same day ChatGPT opened our Services page and listed every price correctly.

A readable page was not enough. The crawler also had to be let in. We added a test for this to our free check that day.

How to test your own site

  1. Look at the page code. In Chrome, Firefox or Edge, right-click the page and choose “View page source”. Search for a sentence from the middle of your page. If it is not there, the text is added by JavaScript.
  2. Run a check that reads the page as a crawler does. Our free AI check requests one page without running JavaScript and shows the words that arrive. It reads your robots.txt, then asks for the page again under the names of the main AI crawlers to see whether your server answers them.
  3. Ask an assistant to open the page. Give it the full address and ask what the page says. If it reports that it could not open the page, or describes something that is not on it, look closer. Assistants can answer from a saved copy. In our test ChatGPT kept showing its saved copy of a page it had opened before. Adding a new ending to the address, such as ?check=1, made it fetch the page again.
  4. Ask your host. If crawlers are refused, send your host the crawler names and ask whether a firewall or bot rule is turning them away.

How to fix it

The page arrives empty

The text has to be in the page code. A developer can switch on server-side rendering or static generation in the framework the site already uses. The other route is to rebuild the pages as static files, which is a service we sell: a static rebuild at €450 setup plus €30 a page, with the same design and content.

The server refuses crawlers

This is a setting at your host or security service, not a fault in your pages. Ask for the AI crawlers you want to be allowed through. If the host cannot do that, the remaining fix is a different host.

robots.txt shuts them out

Look for a rule that names an AI crawler, or all crawlers, followed by Disallow: /. Remove it or narrow it. Our guide to AI crawlers and robots.txt lists the names and what each one is for.

What being readable does not do

Fixing these problems makes your pages available to AI assistants. It does not make an assistant mention or recommend you. That depends on what your pages say, on what other sites say about you and on choices the assistants make. Nobody can promise it.

Being readable is the first condition. Without it, an assistant describes you from whatever else it can find.

Sources

Every figure and quoted fact in this guide was checked against these pages on 17 November 2026. Prices and product rules change, so check the source before you rely on a number.

  1. Vercel and MERJ: The rise of the AI crawlerPublished 17 December 2024. Which crawlers run JavaScript.
  2. Google Search Central: Understand JavaScript SEO basicsHow Google’s crawler renders JavaScript.
  3. Cloudflare: press release of 1 July 2025AI crawlers blocked by default; new domains are asked at sign-up.

FREE CHECK / NO SIGN-UP

Can AI read your site?

Enter one address and see what an AI crawler receives from your page: the words, eight checks and which crawlers are let in.

KEEP READING

More guides

All guides
  1. AI visibility

    AI crawlers and robots.txt: who is who, and how to let them in

    The crawlers of OpenAI, Anthropic, Perplexity and Google by name, what each one is for, and the robots.txt rules that let them in or keep them out.

    5 min read

  2. Running a website

    Website running costs: static, WordPress, Webflow and Framer

    What a small website costs to run each year as static files, on WordPress, Webflow or Framer: fees, hosting, upkeep, and what happens if you leave.

    7 min read