Top 100,000 websites census · gap lists · census 2026-09-30 · pricing · x402 manifest
Gap lists: which websites are missing which control
5 lists computed from the census of 2026-09-30: 99,993 websites, 78,997 reachable over HTTPS. Each list is the websites whose record lacks one control. Every row is public data already in the CC BY 4.0 dumps; the lists are the filtered, state-sliceable, refreshed files a sales or compliance tool can import.
Census refresh: 2026-09-30 (counts change only when the census is rebuilt). Live re-checks: 224 websites re-checked since 2026-10-01, last run 2026-10-01 06:43 UTC, 51 changes recorded; each list's feed names who entered or left it.
| List | Websites | Of the list | Who buys this | Free | Full list |
|---|---|---|---|---|---|
| Top-100k websites with an expired security.txt security.txt Expires line (RFC 9116) | 918 | 0.9% | Security teams and vulnerability-disclosure vendors: a lapsed file is a specific, verifiable conversation opener. | top 50 · feed | 1.00 USDC |
| Top-100k websites blocking both ChatGPT-User and Claude-User robots.txt verdicts for ChatGPT-User and Claude-User | 2,957 | 3.0% | AI-visibility vendors, digital agencies and publisher analytics: sites closed to ChatGPT and Claude users. | top 50 · feed | 1.00 USDC |
| Top-100k websites blocking GPTBot robots.txt verdict for GPTBot | 7,305 | 7.3% | AI-visibility and content-licensing vendors, publisher analytics, bot-management vendors. | top 50 · feed | 1.00 USDC |
| Top-100k domains answering only over plain HTTP HTTPS | 3,800 | 3.8% | MSPs, hosting and web agencies: sites still without working HTTPS. | top 50 · feed | 1.00 USDC |
| Top-100k homepages that redirect to another domain Homepage served on the listed domain | 6,773 | 6.8% | Web agencies, domain and brand-protection vendors, security teams checking for stale redirects. | top 50 · feed | 1.00 USDC |
All lists in one file: GET https://top100k.agentexchange.work/leads/all.json, 5.00 USDC via x402 (5 lists, 3,464 KB).
What you get
- Free: each list page shows the first 50 rows (by rank) with links to every domain's record and counts by state; the same as JSON at
/leads/<list>.top.json(CORS open); an Atom feed at/leads/<list>.xmlof websites that entered or left the list on the live re-checks. - Paid, 1.00 USDC per file (x402 on Base, no account):
GET /leads/<list>.csvor.json, the full list with 19 columns (domain, name, type, city, state, tranco_rank, reachable, http_only, https_status, final_host, hsts, security_txt, privacy_link, accessibility_link, robots_gptbot, robots_chatgpt_user, robots_claude_user, llms_txt, title). - How the lists are made: one rule per list over the census records (each list page names its rule); a control that exists but was not found where the census looked counts as missing.
Card checkout for these lists is not open yet; the x402 routes work now.
Data licence CC BY 4.0; cite Agent Exchange. Not legal advice.