The short version
- AI crawlers such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended fetch the raw HTML a server returns. Most do not run JavaScript, and none can read text that only exists as pixels in a scanned image.
- A fact printed in a scanned PDF with no text layer, or rendered only after a browser executes JavaScript, is invisible to these crawlers even though a person sees it perfectly.
- When the authoritative page is unreadable, the model falls back to weaker sources — old news, third-party explainers, its own training data — and states an approximation with full confidence.
- You can test any page in minutes with a single command, and the fix is unglamorous: publish the fact as machine-readable text, add structured data, and maintain an llms.txt index.
- The winnable insight: the correct official rule frequently exists and is free to read. No machine can extract it. That is a content-format problem, not a policy one.
The fact exists, and the machine still cannot read it
Consider a common situation. A citizen asks an AI assistant about a fee, a deadline or an eligibility rule. The AI assistant answers confidently, and the answer is wrong or out of date. Someone on your team checks and finds that the correct, current rule is published on the official website — it has been there for months, it is free to read, and any person with a browser can find it in seconds. So why did the machine get it wrong?
In a large share of cases the answer is not that the information is missing, hidden behind a login, or wrong on your own site. It is that the fact is not machine-readable. A human eye and an AI crawler do not see the same page. The person's browser does a great deal of work — running scripts, rendering images, interpreting layout — before the fact appears on screen. The crawler skips almost all of that. If the crucial sentence only becomes visible after that work, the crawler never sees it. This is the machine-readability gap, and it is the root cause of a surprising number of confidently wrong government answers.
How an AI crawler actually fetches a page
It helps to be concrete about what these bots do, because the mental model most people carry — that the crawler "visits the page like I do" — is the source of the confusion. When an AI company's crawler fetches one of your URLs, it typically makes a single HTTP request and reads whatever the server sends back in that first response. That response is the raw HTML document. The crawler parses the text and links in that document and, in most cases, stops there.
Your web browser does far more. After receiving the same HTML, it downloads and executes the JavaScript the page references, fetches images and fonts, and lets scripts rewrite the page — inserting text, loading data from an internal service, building tables and panels that were not in the original HTML at all. Only after all of that runs does the finished page appear. The gap between "the HTML the server first sent" and "the page a human finally sees" is exactly where facts disappear for a machine.
Most AI crawlers do not run JavaScript
The main AI-search crawlers are, for the most part, lightweight text fetchers. They request the HTML and read it. They generally do not spin up a full browser engine to execute your scripts, because doing so at web scale is slow and expensive. This is different from Google's traditional search indexer, which does render JavaScript for ranking purposes. The practical consequence for AI answers is stark: if a fact is only written into the page by JavaScript after load, a crawler that reads only the initial HTML will treat your page as though that fact is not there.
- GPTBot and OAI-SearchBot (OpenAI) — fetch and read HTML; do not depend on client-side rendering.
- ClaudeBot and Claude's user-triggered fetcher (Anthropic) — read the server response; content must be present in the returned markup.
- PerplexityBot (Perplexity) — retrieves pages to ground answers; server-rendered text is what it can rely on.
- Google-Extended — a permission control governing use of your content for AI features; the safe assumption is that AI-side consumption favours text already in the HTML.
Two different pages from one URL
A single URL effectively serves two documents: the raw HTML the server returns first, and the fully rendered page a browser builds after running scripts. Humans read the second. Most AI crawlers read the first. Any fact that exists only in the second is, for those crawlers, unpublished.
Failure one: the scanned PDF with no text layer
Government information has a deep tradition of the authoritative PDF — the circular, the gazette notice, the official form, the signed order. Many of these are produced by scanning a printed or signed paper document. A scanned PDF is, underneath, a photograph of a page. The words you read are pixels arranged to look like letters. There is no underlying text the way there is in a document typed and exported directly to PDF.
The distinction that matters is the text layer. A born-digital PDF, or a scan that has been run through optical character recognition (OCR), carries a hidden layer of actual selectable, copyable text behind the image. A raw scan does not. If you open the file and cannot select the text with your cursor — if dragging across a paragraph highlights nothing, or highlights a whole block as an image — then there is no text layer, and a machine cannot extract a single word. The rule may be perfectly legible to your eye and completely invisible to a crawler. The simplest test is to try to select and copy a sentence; if nothing copies, no machine can read it either.
This is why a correct official rule can be simultaneously published and unreadable. The order exists, it is free, it is the authoritative source — and it is a picture. The model, unable to lift the text, reconstructs the rule from whatever it can read: a news summary, an explainer site, an older superseded copy, or its own training data. The confident wrong answer is the model filling a vacuum your own document created.
Failure two: the JavaScript-only portal
The second common failure is the modern web portal. Many public services now sit inside single-page applications: you open the URL, the browser loads a mostly-empty HTML shell, and JavaScript then fetches the real content from an internal service and paints it onto the page. Eligibility rules, fee tables, scheme details and status pages are frequently built this way. For a human it works fine — the content appears after a moment.
For a crawler that does not execute JavaScript, the URL returns an almost-empty shell. Where the fact should be, there is a loading placeholder or an empty container, and the crawler reads exactly that: nothing of substance. The fact never entered the HTML; it was going to be assembled in the browser, and the crawler is not a browser. Once again the authoritative page technically exists, a person can read it, and the machine sees a blank.
"But I can see it" is the trap
The most misleading evidence is your own screen. Because your browser runs the scripts and renders the images, you always see the finished page and naturally conclude the fact is published. The crawler's experience is invisible to you unless you deliberately look at the raw server response. Never judge machine-readability by what your browser shows.
Test whether your fact is in the server HTML
You do not need specialist tooling to find out what a crawler sees. The point is to look at the raw HTML the server returns — before any JavaScript runs — and check whether the fact is actually in it. From a terminal, request the page the way a crawler would, then search the response for the key sentence or figure.
# Fetch the raw HTML a crawler receives (no JavaScript executed),
# identifying as an AI crawler, then search it for your fact.
curl -sL -A "GPTBot/1.0" https://example.gov/scheme/eligibility | grep -i "threshold"
# No match? The fact is not in the server HTML.
# Either JavaScript inserts it later, or it only lives inside a PDF/image.If the command prints the sentence containing your fact, a text-only crawler can read it. If it prints nothing — or prints an empty shell, a loading message, or a link to a PDF but not the fact itself — then the fact is not machine-readable at that URL. For PDFs, the equivalent check is to open the file and try to select the text; if you cannot select and copy it, it is a scanned image with no text layer, and no amount of crawling will extract the words. You can make the HTML check part of a routine review: run it against your highest-stakes pages and treat an empty result as a defect to fix.
A quick way to see the difference on a portal is to compare the raw response with the rendered page. View the page normally in a browser, then view the page source or the raw curl output. If the fact is on screen but absent from the raw HTML, JavaScript is doing the work, and a crawler is missing it.
<!-- What a JS-only portal often returns to a crawler: an empty shell -->
<div id="app"><!-- content loaded by JavaScript after page load --></div>
<!-- What a machine-readable page returns: the fact, in the HTML itself -->
<main>
<h1>Scheme eligibility</h1>
<p>The current income threshold for this scheme is 250,000 per year,
effective 1 April 2026. Applicants at or below this figure qualify.</p>
</main>The fix: restructure the fact as machine-readable content
The remedy is not to abandon PDFs or rebuild every portal. It is to make sure that for each fact people ask about, there is a plain-HTML, server-rendered home for it that a crawler can read on the first request. Three practices carry most of the weight, and none of them require arguing with the AI models.
1. Put the fact in server-rendered HTML text
For every high-stakes fact — a fee, a threshold, an eligibility rule, a deadline, a current status — ensure the authoritative sentence is written as ordinary HTML text that appears in the raw server response. If your portal renders client-side, publish a companion page or use server-side rendering so the sentence is in the initial HTML. If the source of record is a scanned PDF, transcribe the key provisions into an HTML page alongside it, or at minimum run OCR so the PDF gains a real text layer. State the fact in one unambiguous sentence, in the words a citizen would use to ask it, near the top of the page.
2. Add structured data so the fact is explicit
Prose can be lifted, but typed, structured annotation removes ambiguity entirely. Marking up a service, its eligibility and its effective date as machine-readable structured data turns an implicit statement into an explicit one a machine can consume without inference. Always include an effective or last-reviewed date; recency is one of the few signals that reliably helps a model prefer your current page over an older copy. The mechanics are covered in structured data for GEO.
3. Publish an llms.txt index and confirm the crawlers reach it
Beyond individual pages, maintain a consolidated, plainly-structured index of your key current facts and point AI crawlers to it with an llms.txt file. Then verify that the bots can actually fetch your pages — that nothing in your robots rules, firewall or rendering setup is quietly blocking them. AI crawler analytics shows which AI assistants are fetching your content and which are being turned away or served an empty shell.
1
HTTP request most AI crawlers make — they read the raw HTML from that response and generally do not run JavaScript
0
words a machine can extract from a scanned PDF that has no text layer
Free
cost of reading the correct official rule — which is precisely why an unreadable format, not paywalling, is the winnable root cause
Why this is the tractable problem
Many causes of wrong AI answers are genuinely hard to influence: a model's training cutoff, how it weighs competing sources, whether it decides to retrieve live pages at all. The machine-readability gap is different, and that is what makes it worth prioritising. It is entirely within your control, the correct content usually already exists, and the fix is a content-format change rather than a policy change or a negotiation with an AI company.
Put plainly: when the correct rule is published, free, and simply trapped in a format no machine can read, you are one transcription or one rendering change away from letting the models quote your actual position instead of an approximation of it. This is the cheapest, most defensible accuracy win a public body can make. For the wider discipline of keeping official facts current in AI answers, see AI accuracy for government communications; for the format-level detail treated here, the machine-readability gap is where the effort pays back fastest. It is also worth putting on a standing schedule — the models change, portals get rebuilt, and a page that was readable last quarter can silently regress.
None of this replaces authoritative publishing or the signed order as the legal source of record. It extends them to a channel that has quietly become one of the first places people go for facts about their government. Making each official fact readable by a machine — not just visible to a human — is how a public body keeps its own record from being overwritten by a confident guess.