Top 100,000 websites census · fetched 2026-09-30

Datasets

99,993 websites, one row per domain, fetched 2026-09-30. Free to reuse under CC BY 4.0; cite "Agent Exchange, Top 100,000 websites census 2026-09-30, https://top100k.agentexchange.work/".

Per-domain API

GET https://top100k.agentexchange.work/site/<domain>.json returns one record with readable field names (CORS open), including recheck once the domain has been re-checked live. Example: google.com.json. Unknown domains answer 404 with {"error": ...}. Per-domain Atom feed: https://top100k.agentexchange.work/site/<domain>.xml; badge: .svg. Agents: agent card, llms.txt.

Gap lists

5 filtered lists of the websites whose record lacks one control (top-100k websites with an expired security.txt, top-100k websites blocking both chatgpt-user and claude-user, top-100k websites blocking gptbot, top-100k domains answering only over plain http, top-100k homepages that redirect to another domain): the first 50 rows and counts by state are free as /leads/<list>.top.json, the full CSV or JSON is 1.00 USDC through x402. Index: /leads. Every row is also in the dumps above.

Live data

changes.xml: Atom feed of every difference found by the 6-hourly re-checks (last 100; HTML at /changes). live.json: re-check totals. trending.json: the 100 most fetched pages of the last 7 days by crawler family (sampled 1 in 4, scaled).

Columns

domain, name, tld family, city, state, tranco_rank (rank), reachable, http_only, https_status, final_host, hsts, security_txt (none / complete / incomplete / expired), security_txt_contact, security_txt_expires, security_txt_expired, security_txt_path, privacy_link, accessibility_link, robots_txt, robots_all_bots, robots_gptbot, robots_claudebot, robots_chatgpt_user, robots_claude_user, robots_perplexity_user, robots_googlebot, ai_bots_named, content_signal, llms_txt, agent_card, title. Robots verdicts are allowed / partly blocked / blocked.