# Top 100,000 websites census

> Of the top 99,993 Tranco domains, 78,997 answered over HTTPS; 8.9% publish a security.txt; 7,305 block GPTBot and 6,404 block ClaudeBot; 8,873 publish an llms.txt and 58 an A2A agent card. Fetched 2026-09-30.

The 99,993 highest-ranked domains in the Tranco list (list pulled 2026-09-29) were fetched once over HTTPS on 2026-09-30, with robots.txt, security.txt (both locations), llms.txt and the A2A agent card. 78,997 answered with a website. Lists and lookup pages are ordered by Tranco rank.

## Report
- https://top100k.agentexchange.work/: The top 100,000 websites census, September 2026: what the web's most-visited domains publish for security researchers and AI agents. Headline numbers, tables by tld family and by state, named lists, method.
- https://top100k.agentexchange.work/findings.json: all aggregates (population 99,993, fetched 2026-09-30)

## One website
- https://top100k.agentexchange.work/site: lookup (search form; ?domain=<host> redirects to the page)
- https://top100k.agentexchange.work/site/<domain>: HTTPS, HSTS, security.txt, privacy and accessibility links, robots.txt verdicts for GPTBot, ClaudeBot, ChatGPT-User, Claude-User, Perplexity-User and Googlebot, llms.txt, A2A agent card, what to fix
- https://top100k.agentexchange.work/site/<domain>.json: the same record as JSON (CORS open, CC BY 4.0); example https://top100k.agentexchange.work/site/google.com.json

## Datasets
- https://top100k.agentexchange.work/top100k-web-census-2026-09.csv: One row per domain (99,993 rows), readable columns (CSV, 15.3 MB, CC BY 4.0)
- https://top100k.agentexchange.work/findings.json: Aggregates: overall, by tld family, by state, named lists (JSON, 335 KB, CC BY 4.0)

## Machine access
- https://top100k.agentexchange.work/.well-known/agent-card.json: A2A agent card describing the JSON lookup
- https://top100k.agentexchange.work/sitemap.xml

## Related
- https://grade.agentexchange.work/reports/ai-crawler-census-2026-09: AI crawler census (top 10,000 sites)
- https://gov.agentexchange.work/: Government web census (.gov)
- https://banks.agentexchange.work/: US bank web census
- https://colleges.agentexchange.work/: US college web census
- https://schools.agentexchange.work/: US school district web census
- https://libraries.agentexchange.work/: US public library web census
- https://agentexchange.work: Agent Exchange