If the goal of the web was just to transmit objective facts like 'who created Linux', this wouldn't be a problem at all. Wikipedia handles that pretty dece.

The issue arises when we move away from objective trivia and into subjective, localized buying intent, which is where the web monetizes itself.

If I ask an LLM 'who created Linux?', there is one right answer. But if I ask an LLM 'who are the best commercial roofers in Houston?', there isn't one right answer. There are dozens of highly qualified local businesses that do possess unique value, unique pricing, and unique availability.

When an LLM answers that roofing question, it typically cites 3 to 5 businesses. The other 40 legitimate roofing companies in the area are left out. My study isn't arguing that every single one of those 40 companies deserves to be in the answer; it's pointing out that those 40 companies currently have no idea they are being left out.

In the Google era, if you weren't on page 1, you could look at Search Console, see your ranking, check your backlinks, and understand why. In the AI Search era, businesses are being scraped to build these answers, but they have zero telemetry on whether they are actually making the cut. This index is just an attempt to provide that missing telemetry.

The question is very important. What if you ask/tell an AI "I need a 2 bedroom vacation rental in Park City for a trip this fall". If you only get Airbnb results you are missing a lot of the true answer.

This is not a complaint.

I think your assessment of this result being as expected... but this is about the LLM equivalent of SEO becoming an area of interest

I wouldn't even consider it a complaint. For a builder, I would consider it an opportunity...

I just asked ChatGPT-5.6 that question, no source was given.

Not even Wikipedia.

It isn't a complaint. It's an ad. This is literally an ad for some sort of "get cited by AI" service.

Its a marketing post for a tool that offers visibility to site owners. Honestly these type of "neutral data reports" masquerades should be flagged or adequately disclosed.

LLM's were obviously going be ruined by two factors:

1- LLM Crawler Optimization: the new SEO

2- Weightings-For-Pay: for a fee have your product or service come up more frequently in associated answers

> Are we saying that it's now a problem that we're not getting scraped?

No, it is saying that even the ones that are being scraped are not being included in results

I get that pushback. My goal with this index/report wasn't to write a new 'AI SEO' playbook or encourage people to start gaming the system. The goal was simply to shine a bit of light on what's going on. Right now, there is a massive information asymmetry: AI companies are turning their assistants into primary search engines, but webmasters have zero visibility into whether their sites are actually being cited in those live answers.

You're absolutely right to be concerned about an arms race. If the data showed that doing X, Y, and Z guaranteed a citation, we'd be right back to the worst days of keyword stuffing.

But what this data actually shows is the opposite: even if you do everything 'right' (allow the retrieval bots, provide perfect machine-readable schema, don't block anything), you still have a 94.8% chance of never being cited. How LLMs cite is entirely opaque.

I built this to give site owners a baseline measurement of what is actually happening to their content today (as a free, anonymous view), so they can make an informed decision on whether keeping their doors open to these crawlers is actually worth it.

The problems with micropayments are

1. The market for lemons. In fact sites trying to "monetize their content" are the most likely to be lemons, so just asking is a signal that your "content" is not worth anything.

2. If your information does have value, it competes in a market with other sites full of high quality information that aren't trying to monetize it, meaning your specific information has to be specifically very valuable and not available yet for free elsewhere. If it is valuable as information (i.e. not something like creative writing), it will quickly spread and become freely available.

Lots of "content" just isn't worth anything (or has negative worth: it wastes your time). e.g. consider youtubers begging to get viewers to like/subscribe to increase their reach, and people generally don't despite it costing them nothing. Because it's not even worth a click to them.

Seems like the opposite if the problem you’re trying to solve is “my site isn’t cited over someone else’s”. HTTP 467 - pls cite me, I’ll pay you.

For standard responses you're not wrong, but increasingly there are a lot of LLMS that are essentially doing RAG against search results. For example this is I believe how Kagi works, and Google AI overviews. I have increasingly seen Claude and chat GPT also doing the same thing where instead of answering a question from the knowledge Bank they will do a web search and cite the responses that were used.

You can do it the same way human does: recall a fact from memory, and then search to identify a citation. The citation isn't "here's why I think this" but instead "here's where you can verify this".

This is why it's dumb when inferencing with a model to ask "...and cite all sources". Total waste of time and likely to make the response worse.

BUT

On models with web search, they can scan the sources in context, and those are pretty good at correct citations. However it's still a "trust, but verify" situation.

> It is perfectly common to find that a "source" does not contain the statements.

It is commonly this way right now, but it doesn't have to stay this way forever.

It is also common, in my usage at least, to iteratively brow-beat the bot into paring its statements down to those that which are supportable by its sources. Doing so just takes repetition, and that repetition takes time and burns more tokens.

With the present state of things, the prompts to get moving on this and to guide the ultimate response into something that is verifiably supportable by outside sources can usually be simple and largely generic.

They're easy enough prompts that a subagent can produce them.

(I've done it myself with Codex subagents and it worked very well, aside from the unsustainable burn rate that did not fit my budget.)

This is true, but good LLMs (even local, consumer-sized ones these days) can do this much better than … whatever it is that Google’s AI overviews use.

Think about your coding harness: the model reads a bunch of files into the context, and it generally doesn’t forget/hallucinate which lines came from which file.

Then the least they can do is cite ALL their sources. At least somewhere on their website.

Are you sure? Because this sounds like GPT3-era understandings of LLM-isms

Lol, this same behavior comes up more often than you'd expect in academia too. Many many citations that nobody cross checks that also don't support the originally cited piece of information.

> It is architecturally impossible for an LLM to associate a link or citation that it crawled with a response that comes out the other end.

I mean, its not. Lots of LLM's focused on finance do this already.

Bloomberg's own ASKB produces results and provides links back to the source documents or urls that it references so people who care about correctness can verify the results.

99% of AI users these days aren't using a 'raw' LLM. They are interacting with a harness that includes tools to do search of the live (or recently crawled) web. And so the output users actually see could absolutely include correct citation of sources. So the 'architecturally impossible' bit may be technically correct but it is not practically relevant. Now whether those harnesses do a good job of orchestrating LLM output to get accurate citations (and whether they are transparent about their process) is another thing entirely. But if you're contemplating 'what LLM's can do' and aren't taking into account the harness and tooling ecosystem they are embedded in then you're missing the point.