JavaScript and AI crawlers: how to check what ChatGPT sees

A crawler is a program that fetches pages automatically. Rendering here means that it runs a page’s scripts and reads the result. The evidence is thin and varies in reliability, so the table gives the type of source and its date for each system. The numbers in the “Source” column refer to the list of sources at the end of this article.

The server logs come from a single news site. Requests there were counted by user agent, the name a program uses to identify itself to the server, and the analysis comes from a vendor of log tools (Forge AI Bot Log, Sep 24, 2026). The 2024 measurement was published by the cloud platform Vercel with MERJ; Microsoft Copilot is missing from it because it doesn’t use a user agent of its own (Vercel and MERJ). The overview comes from searchVIU, a vendor of SEO tools (searchVIU, Nov 3, 2026). We haven’t found a newer measurement of comparable size. JavaScript isn’t mentioned in the crawler documentation from OpenAI or Perplexity. We also found no statement from OpenAI on the reports that have appeared since September 2026.

Why it matters: Whatever only appears in the browser is missing for systems that don’t run JavaScript. On the news site, ChatGPT-User fetched a page with fuel prices 157 times in 30 days, and Claude-User 80 times. Neither saw a price, because scripts inserted the prices. Googlebot does render JavaScript, so the gap often goes unnoticed in Google Search. How Google handles this is covered in our guide to technical SEO, in the section Can Google see content that JavaScript generates?

Agents that drive a real browser do run scripts. In your logs, they often look like ordinary visitors using Chrome (Forge AI Bot Log). This how-to doesn’t apply to such agents.

Done when you know which of these systems matter for your customers.

Why: A crawler without JavaScript gets exactly this raw HTML. Your browser shows the finished page after all scripts have run, which is why you won’t spot a gap while browsing. The check has three steps: this request first, then the comparison with the browser, and finally finding the cause.

How to do it:

1. In your browser, copy a sentence your customers should read, ideally one that names your service and location. If you can, pick a stretch without umlauts, accented letters or typographic apostrophes.
2. Put the URL and the sentence into the command for your system.
3. Read the output: 1 or more means the sentence is in the raw HTML. grep counts lines, not matches. If you get 0, first check whether the request worked (the third step of this check).

curl -sL https://www.example.com/services/ | grep -c "Your key sentence"

(curl.exe -sL https://www.example.com/services/ | Select-String -SimpleMatch "Your key sentence").Count
(Invoke-WebRequest -UseBasicParsing -Uri https://www.example.com/services/).Content.Contains("Your key sentence")

The first PowerShell line prints a number, the second True or False. Windows 10 and 11 ship with curl (curl project). Type curl.exe there, because curl alone is an alias for Invoke-WebRequest in PowerShell 5.1 (Microsoft on Invoke-WebRequest). That cmdlet’s -UseBasicParsing switch has been necessary since a security update on December 9, 2025. Without it, PowerShell asks on every request whether script code from the page may run.

Common mistakes:

• Non-ASCII characters on Windows. With curl.exe and Select-String, the search found nothing in our test on November 9, 2026, as soon as the sentence contained a German umlaut, an accented letter or a typographic apostrophe. The console reads the response with an older character set (code page 850 on our German Windows machine). The Invoke-WebRequest line found the text. If you want to stick with curl.exe, first set [Console]::OutputEncoding = [Text.Encoding]::UTF8.
• Line break in the source. If the sentence is split across two lines in the HTML, grep won’t find it. Use three or four words, or the script in the next section.
• Text only in data. If the sentence appears only in a data block, such as JSON for the application, grep counts it even though it isn’t on the page without JavaScript. Vercel and MERJ consider it possible that AI models still make use of such data. Don’t rely on it.

Example: On a test page of our own built entirely on the client side, grep -c returned 0. On the static version of the same page, it returned 1.

Done when you know for every important page whether its key sentence is in the raw HTML.

Our script compares both versions for the whole page and lists every text block that exists only in the DOM. The --dump-dom flag prints the page after Chrome has run its scripts (Chrome documentation). With --virtual-time-budget, content that loads later gets eight seconds of virtual time.

How to do it: Save the script as raw-vs-dom.py. You need Python 3 and Chrome or Edge. Run it with python on Windows and python3 on macOS and Linux, followed by the URL and your key sentence. Our code examples are available under the MIT-0 license.

#!/usr/bin/env python3
"""Compares the text in the delivered HTML with the text after JavaScript has run.

Usage: python3 raw-vs-dom.py https://www.example.com/page/ "a sentence from your page"

Requires Python 3 and Chrome or Edge. License: MIT-0 (datafalke.de).
"""
import html.parser
import shutil
import subprocess
import sys
import tempfile
import urllib.error
import urllib.request

BROWSERS = [
    r"C:\Program Files\Google\Chrome\Application\chrome.exe",
    r"C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe",
    "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome",
    "google-chrome", "chromium", "chromium-browser",
]
USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36"
NO_TEXT = {"title", "script", "style", "template"}
BLOCK_TAGS = {"p", "li", "h1", "h2", "h3", "h4", "h5", "h6", "td", "th", "div", "section", "main",
              "article", "header", "footer", "nav", "aside", "form", "tr", "pre", "blockquote", "figcaption"}


class Text(html.parser.HTMLParser):
    def __init__(self):
        super().__init__()
        self.skipping, self.blocks, self.scripts, self.data_blocks = 0, [[]], 0, 0

    def handle_starttag(self, tag, attrs):
        if tag == "script":
            kind = dict(attrs).get("type") or ""
            if kind in ("", "text/javascript", "application/javascript", "module"):
                self.scripts += 1
            else:
                self.data_blocks += 1
        if tag in NO_TEXT:
            self.skipping += 1
        if tag in BLOCK_TAGS:
            self.blocks.append([])

    def handle_endtag(self, tag):
        if tag in NO_TEXT and self.skipping:
            self.skipping -= 1

    def handle_data(self, data):
        if not self.skipping:
            self.blocks[-1].append(data)


def normalize(text):
    return " ".join(text.replace("\u00ad", "").split())


def parse(source):
    text = Text()
    text.feed(source)
    blocks = [normalize("".join(block)) for block in text.blocks]
    return [block for block in blocks if block], text.scripts, text.data_blocks


def raw(url):
    request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
    try:
        response = urllib.request.urlopen(request, timeout=30)
    except urllib.error.HTTPError as error:
        response = error
    charset = response.headers.get_content_charset() or "utf-8"
    return response.getcode(), response.read().decode(charset, "replace")


def dom(url):
    browser = next(filter(None, map(shutil.which, BROWSERS)), None)
    if browser is None:
        sys.exit("Chrome or Edge not found: add the path to BROWSERS.")
    profile = tempfile.mkdtemp()
    try:
        result = subprocess.run(
            [browser, "--headless", f"--user-data-dir={profile}", "--virtual-time-budget=8000", "--dump-dom", url],
            capture_output=True, timeout=120)
    finally:
        shutil.rmtree(profile, ignore_errors=True)
    return result.stdout.decode("utf-8", "replace")


def main():
    url = sys.argv[1]
    sentence = normalize(sys.argv[2]) if len(sys.argv) > 2 else None
    status, source = raw(url)
    raw_blocks, scripts, data_blocks = parse(source)
    dom_blocks, _, _ = parse(dom(url))
    raw_text, dom_text = " ".join(raw_blocks), " ".join(dom_blocks)
    only_in_dom = [block for block in dom_blocks if block not in raw_text]
    print("Status:      ", status)
    print("Scripts:     ", scripts, "executable,", data_blocks, "data block" if data_blocks == 1 else "data blocks", "(such as JSON-LD)")
    print("Words:       ", len(raw_text.split()), "in raw HTML,", len(dom_text.split()), "in DOM")
    if sentence:
        print("Key sentence:", "raw HTML", "yes" if sentence in raw_text else "no",
              "| DOM", "yes" if sentence in dom_text else "no")
    print("Only in DOM: ", len(only_in_dom), "text block" if len(only_in_dom) == 1 else "text blocks")
    for block in only_in_dom[:5]:
        print("  -", block[:90])


if __name__ == "__main__":
    main()

What to expect: The table shows the output for five test pages of our own on our test machine. Static means prebuilt files; server-side rendering (SSR) means the server builds the HTML on every request; client-side rendering (CSR) means JavaScript builds the page in the browser (web.dev on rendering types). On real websites, the numbers are larger, but the pattern holds.

Measured Nov 9, 2026 · curl 8.21.0, GNU grep 3.0, Python 3.12, Chrome 156, fetches of five test pages of our own with Python and curl, output of the comparison script

How to read the rows:

• Static: The text is readable by every crawler.
• Server-side: The text is readable by every crawler; scripts only make the page interactive.
• Mixed: The paragraph added by the script is missing without JavaScript.
• Client-side: The text is missing without JavaScript; the 3 words are a notice in the noscript element.
• Data only: grep counts the sentence, but it only appears on the page after the script runs.

To a crawler, static and server-rendered pages look the same: Both pass the check. If the server rebuilds the page on every request, a block with changing values such as a time can show up as “only in DOM,” because the script fetches the page twice. That isn’t a rendering error. This is what the output looked like for the mixed test page:

Status:       200
Scripts:      1 executable, 0 data blocks (such as JSON-LD)
Words:        25 in raw HTML, 35 in DOM
Key sentence: raw HTML yes | DOM yes
Only in DOM:  1 text block
- Maintenance contracts are available for systems up to 100 kWp.

Without Python, you can count the key sentence in the DOM directly. Point Chrome to a separate profile folder so that an open Chrome window doesn’t get in the way:

google-chrome --headless --user-data-dir=/tmp/df-probe --dump-dom https://www.example.com/services/ | grep -c "Your key sentence"
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" --headless --user-data-dir=/tmp/df-probe --dump-dom https://www.example.com/services/ | grep -c "Your key sentence"

$chrome = "C:\Program Files\Google\Chrome\Application\chrome.exe"
Start-Process $chrome -ArgumentList "--headless", "--user-data-dir=$env:TEMP\df-probe", "--dump-dom", "https://www.example.com/services/" -NoNewWindow -Wait -RedirectStandardOutput dom.html
(Select-String -Path dom.html -SimpleMatch "Your key sentence" -Encoding utf8).Count

On Windows, write the DOM to a file first, because in our test on November 9, 2026, piping it directly with | returned nothing.

Common mistake: The Elements tab in the browser’s developer tools shows the DOM, meaning the page after the scripts have run. The source the server sent is in the “View page source” view (Ctrl+U, on a Mac Command+Option+U). Our how-to on getting recommended by ChatGPT describes this quick check without a terminal.

Done when the “Only in DOM” line shows 0 for your most important pages or lists only text that means nothing to your customers.

Request the page once normally and once with a crawler’s user agent, then read your robots.txt:

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" https://www.example.com/services/
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://www.example.com/services/
curl -s https://www.example.com/robots.txt | grep -i -A1 "gptbot"

curl.exe -s -o NUL -w "%{http_code} %{size_download}\n" https://www.example.com/services/
curl.exe -s -o NUL -w "%{http_code} %{size_download}\n" -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://www.example.com/services/
curl.exe -s https://www.example.com/robots.txt | Select-String -Pattern "gptbot" -Context 0,1

Each of the two request lines prints the response status and size in bytes. The user agent in the example is the one OpenAI specifies for its GPTBot crawler (OpenAI’s crawler documentation). The curl manual describes the -A and -w options. If even the first request returns 403, your CDN probably blocks curl in general. In that case, use -A to send a browser’s user agent.

On our test page with a block by user agent, the normal request returned 200 317 and the request with GPTBot’s user agent 403 92. The test page with a block in robots.txt answered both requests with 200 317. So the request with the user agent doesn’t show the block, but the file does.

Why it matters: From the outside, all of these causes look the same, because AI services don’t have your text. They get fixed in different places: in the website’s code, in robots.txt, or in the settings of your firewall and CDN (the network that delivers your pages). Our article AI crawlers and robots.txt lists which user agents exist and what you should allow. How to check your firewall and CDN is covered there in the section on firewalls and CDNs.

Common mistake: A status 200 with GPTBot’s user agent gets read as proof that GPTBot may read the page. But curl doesn’t check robots.txt.

Done when you know the cause for every page that lacks its key sentence.

1. Change the rendering. For applications built in the browser, server-side rendering or generating pages at build time helps. Google considers this worthwhile because it makes websites faster and because not all bots can run JavaScript (Google’s JavaScript SEO basics). Our comparison Static site vs. WordPress compares which approach suits your website.
2. Move late-loading parts into the HTML. Prices, services, business hours and answers to common questions belong in the initial HTML. If a widget has to stay, state the essentials next to it as text. Microsoft says that content that can’t be rendered reliably may not be indexed or used as a source for answers (Bing’s guidelines).
3. Don’t build separate pages just for crawlers. Google calls a separate version for recognized bots dynamic rendering. It describes that as a workaround that adds complexity and recommends the approaches in point 1 (Google on dynamic rendering).
4. Check again. Repeat the three steps of the check for every page you changed.

That’s why, on the websites we build, the text is in the HTML from the start.

Common mistake: The home page gets switched over while prices and services on subpages keep loading via script.

Done when the “Only in DOM” line no longer lists any important content on the pages you changed.

• Blocks by IP address or country. Your request with a crawler’s user agent comes from your own network. You can identify genuine crawlers by the operator’s IP address. How to count fetches is covered in our how-to on tracking AI traffic.
• Timing. Crawlers can change without anyone announcing it, as the reports from September 2026 suggest. Repeat the check after every redesign and every six months.
• The effect. Once your text is in the HTML, one hurdle is gone. Whether AI answers mention you more often as a result is something this check can’t show. You measure that with a fixed set of questions, as our how-to on measuring AI visibility describes.

Done when your report states what it doesn’t cover.