Most of the web is unbuyable by an agent

Method, in one line: once a week we fetch about nineteen public documents from each domain in a fixed cohort — robots.txt, llms.txt, the sitemap, a handful of /.well-known/ paths and the homepage — score them 0–100 with the same engine our customers pay for, and publish the distribution.

We follow no links, submit no forms, create no carts, use no credentials or cookies, honour robots.txt, and pause between domains. The whole weekly pass is a few hundred requests — less traffic than one person browsing. Sites that cannot be reached are excluded, never scored zero.

Opt out: email [email protected] and the domain is removed from the cohort. No justification needed, and we do not ask for one.

The machine-readable version of this page is at /md/index-report.md →.

Akshay's worry about publishing an index is the obvious one: it is very easy to turn a measurement into a stick, and a league table of named companies scoring badly is a stick. So this one names almost nobody. The cohort is public household-name websites, the seed list is not published, and a site appears by name only if it is ours or if it paid for a certification and therefore chose to be listed.

What is left is the honest part: a distribution. The finding is not that any particular company is behind — it is that the whole cohort is, in the same place, in the same way, and that almost nobody has noticed yet because the websites look fine to a human. That is worth saying out loud, and it does not require naming anyone to say it.

We are in this table too, at whatever we actually score. Our dental clinic is legible to agents and is currently not buyable by one either. We publish that rather than quietly leaving ourselves out.