How an AI assistant finds, reads and cites a web page

"The AI doesn't know about us" is usually four different problems wearing one coat. Splitting the pipeline into its steps tells you which one you actually have.

Four steps, not one

1. 01Crawl. A bot requests your URL and receives HTML. If robots.txt disallows that bot, or your server returns an error or a challenge page, the pipeline ends here.
2. 02Index. The content is stored and processed — commonly split into passages and embedded — so it can be matched against future questions.
3. 03Retrieve. At question time, a small set of documents is pulled from that index, or fetched live from a search API.
4. 04Generate. A model writes an answer with those documents in its context, and the product decides which of them to display as citations.

Steps 1 and 2 are things you can influence directly. Step 3 you influence indirectly, through how findable and how on-topic your page is. Step 4 is the product's decision, not yours — and no provider publishes the rule.

Three kinds of bot, three different consents

It is easy to talk about "AI crawlers" as one thing. The operators do not treat them as one thing, and the robots.txt tokens are separate on purpose:

The distinction matters because the three choices have genuinely different consequences, and a single blanket Disallow collapses all three into the most restrictive one. The next article lists the specific user-agents.

Rendering: if the text needs JavaScript, assume it may be missed

Google renders JavaScript, but does so in a second pass with its own queue and cost. Many retrieval fetchers are simpler: an HTTP request, then a parse of whatever HTML came back. Text that only exists after client-side hydration may never be seen by them.

You do not need to guess how any particular fetcher behaves. Ask your server what it hands over before a browser touches it:

curl -sL https://example.com/your-page | sed -e 's/<[^>]*>//g' | head -60

If the paragraph you want quoted is not in that output, it is not guaranteed to reach anything. Server-rendered or statically generated HTML removes the question entirely.

What makes a passage quotable

Once you are retrieved, your page is competing for space inside an answer that has to be short. Passages that survive that compression tend to share the same properties — and they are properties of editing, not of tricks:

• The answer arrives in the first one or two sentences under a heading, not in the conclusion.
• Paragraphs are self-contained: no "as mentioned above", no pronouns pointing at something three sections back.
• Entities are named explicitly — the company, the product, the place — instead of being carried by context.
• Numbers travel with their unit, their date and their source, so the fact stays checkable once it is lifted out.
• One idea per paragraph, so a retriever can select a passage without dragging in an unrelated half.
• Headings phrased the way a person would ask the question, rather than as clever section titles.

Where to look for evidence

Two sources of real data are available to any site owner, without paying for a tool:

• Server logs. Which AI user-agents requested which URLs, how often, and what status code they got. This is the only direct evidence that step 1 is working.
• Referral traffic in analytics. Visits arriving from assistant domains, which is direct evidence that a citation was shown and clicked.

Neither tells you how often you were used without being cited. That number is not published by anyone, and anybody who reports it to you is estimating.