Census fetched 2026-09-30 · population: Tranco top sites list, top 100,000 domains (list pulled 2026-09-29) · datasets (CC BY 4.0)

The top 100,000 websites census, September 2026: what the web's most-visited domains publish for security researchers and AI agents

The 99,993 highest-ranked domains in the Tranco list (list pulled 2026-09-29) were fetched once over HTTPS on 2026-09-30, with robots.txt, security.txt (both locations), llms.txt and the A2A agent card. 78,997 answered with a website. Lists and lookup pages are ordered by Tranco rank.

Reachable over HTTPS

79.0%

78,997 of 99,993 websites

security.txt

8.9%

7,061 of reachable sites; 4,092 complete and current

Accessibility statement linked

6.7%

5,293 homepages

Privacy policy linked

30.3%

23,928 homepages

Block GPTBot

7,305

4,642 block ChatGPT-User; 6,404 ClaudeBot

llms.txt

8,873

58 publish an A2A agent card

What we found

security.txt: 7,061 of 78,997 reachable sites (8.9%) publish one; 4,092 are complete and current (a Contact line and an unexpired Expires line, RFC 9116); 918 have expired; 742 serve it only at /security.txt instead of /.well-known/security.txt.
Accessibility: 6.7% of reachable sites link an accessibility statement from the homepage.
Privacy: 30.3% link a privacy policy from the homepage.
HTTPS: 78,997 of 99,993 domains (79.0%) answered over HTTPS; 3,800 answer only over plain HTTP; 46.0% of reachable sites send HSTS; 6,773 homepages redirect to a different domain.
AI crawlers and assistants: 59,542 sites serve a robots.txt; 7,305 block GPTBot, 6,404 ClaudeBot and 4,642 ChatGPT-User; 2,241 disallow every bot; 2,957 block both ChatGPT-User and Claude-User; 3,756 block GPTBot while leaving OAI-SearchBot allowed; 14,912 name an AI crawler explicitly; 1,303 carry a Content-Signal line.
Machine-readable surfaces: 8,873 publish an llms.txt and 58 an A2A agent card; 26,647 homepages carry schema.org structured data.

By tld family

TLD family	In list	Reachable	security.txt	valid	privacy link	accessibility link	HSTS	blocks GPTBot	blocks ChatGPT-User	llms.txt	agent card
com	45,411	36,850 (81.1%)	3,067 (8.3%)	1,609	35.1%	6.2%	46.7%	3,445	2,235	5,147	25
other	26,852	21,414 (79.7%)	2,269 (10.6%)	1,440	23.6%	7.0%	46.3%	1,910	1,199	1,959	19
net	6,272	3,581 (57.1%)	260 (7.3%)	153	28.3%	2.9%	35.5%	353	262	263	1
org	4,512	4,001 (88.7%)	318 (7.9%)	220	39.9%	9.3%	45.7%	365	214	295	2
RU	3,578	2,601 (72.7%)	82 (3.2%)	20	16.9%	0.6%	30.4%	184	123	157	0
DE	2,087	1,808 (86.6%)	407 (22.5%)	281	11.4%	3.5%	62.7%	302	147	138	0
JP	1,954	1,587 (81.2%)	35 (2.2%)	22	49.0%	6.8%	35.9%	143	104	81	0
IO	1,868	1,352 (72.4%)	137 (10.1%)	79	41.9%	2.4%	53.3%	65	53	324	8
BR	1,464	1,260 (86.1%)	71 (5.6%)	50	4.4%	1.3%	46.9%	57	35	155	2
CN	1,429	727 (50.9%)	24 (3.3%)	12	7.7%	1.4%	29.0%	28	22	27	0
UK	1,398	1,140 (81.5%)	169 (14.8%)	100	44.4%	29.4%	61.9%	161	67	87	0
IN	1,288	979 (76.0%)	28 (2.9%)	17	27.3%	5.1%	49.3%	49	35	138	0
FR	1,008	902 (89.5%)	131 (14.5%)	58	11.4%	25.9%	56.8%	117	70	42	0
IT	872	795 (91.2%)	63 (7.9%)	31	46.2%	19.6%	51.4%	126	76	60	1

Named lists

Sites telling ChatGPT-User and Claude-User not to open pages for users (first 60 of 300)

facebook.com · gstatic.com · instagram.com · mail.ru · fbcdn.net · twitter.com · dzen.ru · googlevideo.com · amazon.com · doubleclick.net · x.com · pinterest.com · tiktok.com · wa.me · myfritz.net · msn.com · baidu.com · chatgpt.com · yandex.ru · reddit.com · amazonvideo.com · google-analytics.com · registrar-servers.com · nytimes.com · cnn.com · t.co · imdb.com · skybridge.click · amazon.co.uk · naver.com · bloomberg.com · amazon.de · amazonalexa.com · sohu.com · cnbc.com · quora.com · amazon.co.jp · gosuslugi.ru · amzn.to · amazon.co.za · usatoday.com · threads.com · telegraph.co.uk · amazon.fr · amazon.ca · sharethrough.com · primevideo.com · yelp.com · amazon.in · hihonorcloud.com · dailymail.com · dailymail.co.uk · fb.com · amazon.es · pinimg.com · cnet.com · amazon.it · quantserve.com · creativecdn.com · crwdcntrl.net

Sites blocking GPTBot (OpenAI's training crawler) (first 60 of 300)

gstatic.com · instagram.com · mail.ru · twitter.com · dzen.ru · googlevideo.com · amazon.com · whatsapp.net · doubleclick.net · x.com · pinterest.com · tiktok.com · whatsapp.com · yahoo.com · spotify.com · wa.me · myfritz.net · msn.com · baidu.com · chatgpt.com · vimeo.com · yandex.ru · reddit.com · google-analytics.com · registrar-servers.com · snapchat.com · gravatar.com · nytimes.com · flickr.com · soundcloud.com · medium.com · cnn.com · ebay.com · t-online.de · forbes.com · bbc.com · sciencedirect.com · bbc.co.uk · t.co · canva.com · imdb.com · launchpad.net · reuters.com · nature.com · alibaba.com · skybridge.click · amazon.co.uk · naver.com · bloomberg.com · amazon.de · espn.com · globo.com · go.com · claude.ai · amazonalexa.com · yahoo.co.jp · aol.com · pixabay.com · goodreads.com · sohu.com

Homepages that redirect to another domain (first 60 of 300)

amazonaws.com → aws.amazon.com · fbcdn.net → www.facebook.com · twitter.com → x.com · dzen.ru → sso.passport.yandex.ru · office.com → m365.cloud.microsoft · hicloudcam.com → www.ezviz.com · azure.com → azure.microsoft.com · whatsapp.net → www.whatsapp.com · fastly.net → www.fastly.com · doubleclick.net → marketingplatform.google.com · youtu.be → www.youtube.com · skype.com → teams.live.com · googledomains.com → domains.google · googlesyndication.com → www.google.com · wa.me → api.whatsapp.com · zoom.us → www.zoom.com · workers.dev → www.cloudflare.com · cloudflare-dns.com → one.one.one.one · nginx.com → www.f5.com · windows.com → www.microsoft.com · t.me → telegram.org · blogspot.com → www.blogger.com · actify.nl → www.zilverenkruis.nl · b-cdn.net → bunny.net · bit.ly → bitly.com · amazonvideo.com → www.primevideo.com · google-analytics.com → marketingplatform.google.com · outlook.com → outlook.live.com · github.io → pages.github.com · unity3d.com → unity.com · discord.gg → discord.com · xiaomi.com → www.mi.com · lencr.org → letsencrypt.org · adobe.io → developer.adobe.com · adtrafficquality.google → www.google.com · one.one → www.one.com · vkuserphoto.ru → ps.userapi.com · hosting24.com → www.hostinger.com · macromedia.com → www.adobe.com · paytmfirstgames.com → www.firstgames.in · telekom.net → www.telekom.com · rubiconproject.com → www.magnite.com · vungle.com → liftoff.ai · pages.dev → www.cloudflare.com · comcast.net → www.xfinity.com · meraki.com → www.cisco.com · gmail.com → accounts.google.com · wixsite.com → www.wix.com · crashlytics.com → firebase.google.com · mzstatic.com → www.apple.com · telegram.me → telegram.org · googleblog.com → blog.google · aliyun.com → www.alibabacloud.com · mozilla.com → www.firefox.com · onelink.me → www.appsflyer.com · kwai-pro.com → hotvideo.kwai.com · timeweb.ru → timeweb.com · aliexpress.com → www.aliexpress.us · skyhigh.cloud → www.skyhighsecurity.com · kwaipros.com → hotvideo.kwai.com

Sites with a Content-Signal line in robots.txt (first 60 of 200)

cloudflare.com · sentry.io · workers.dev · hosting24.com · zoom.us · pages.dev · amplitude.com · linktr.ee · launchpad.net · hostinger.com · forter.com · nvidia.com · bitdefender.net · trendmicro.com · vercel.app · oxylabs.io · unesco.org · patreon.com · intercom.io · zoom.com · netlify.app · theatlantic.com · supabase.co · corriere.it · usercentrics.eu · bbb.org · mydramalist.com · psu.edu · twilio.com · otomoto.pl · dagbladet.no · segment.io · therapservices.net · wpastra.com · twkan.com · imgsrc.ru · scan-manga.com · intercom.com · bitdefender.com · pewresearch.org · moodle.org · hover.com · livechatinc.com · vinted.fr · segment.com · ninjarmm.com · yotpo.com · metamask.io · laravel.com · onrender.com · customer.io · cookiebot.com · sendbird.com · gameanalytics.com · coinmarketcap.com · railway.app · redis.io · cookieyes.com · padlet.com · vinted.com

Largest websites without an accessibility link on the homepage (first 60 of 300)

google.com · cloudflare.com · facebook.com · gstatic.com · googleapis.com · amazonaws.com · youtube.com · instagram.com · mail.ru · fbcdn.net · dzen.ru · linkedin.com · googlevideo.com · googletagmanager.com · live.com · hicloudcam.com · amazon.com · azure.com · github.com · wikipedia.org · appsflyersdk.com · whatsapp.net · bing.com · fastly.net · googleusercontent.com · doubleclick.net · netflix.com · gandi.net · youtu.be · digicert.com · wordpress.org · skype.com · icloud.com · pinterest.com · tiktok.com · whatsapp.com · goo.gl · cloudflare.net · googledomains.com · yahoo.com · googlesyndication.com · roblox.com · endpoints.news · spotify.com · adobe.com · wa.me · myfritz.net · msn.com · ntp.org · qq.com · zoom.us · nginx.org · workers.dev · baidu.com · chatgpt.com · cloudflare-dns.com · tiktokv.com · vimeo.com · openai.com · opera.com

What websites are expected to publish

robots.txt (RFC 9309): The Robots Exclusion Protocol lets a site tell crawlers what to fetch; it is voluntary. The census records the verdict for GPTBot, ClaudeBot, ChatGPT-User, Claude-User, Perplexity-User and Googlebot and whether any AI crawler is named explicitly. Source
security.txt (RFC 9116): A file at /.well-known/security.txt with Contact and Expires fields tells researchers where to report vulnerabilities. The census marks a file complete when both fields are present and the Expires date has not passed. Source
llms.txt: A proposed Markdown file at /llms.txt that summarizes a site for AI assistants and links its most useful pages. The census counts a file that returns 200, is not HTML and starts with a Markdown heading. Source
A2A agent card: The Agent2Agent protocol publishes a JSON card at /.well-known/agent-card.json describing an agent's skills and endpoint. The census counts a card that parses as JSON and carries a name or skills. Source
Content Signals: A robots.txt extension (Content-Signal line) that states whether content may be used for search, AI input and AI training. The census counts robots.txt files that carry the line. Source
EU Digital Services Act: point of contact and terms: Providers of intermediary services offered in the EU must designate a single point of contact for users and authorities and publish clear terms and conditions (Regulation (EU) 2022/2065, Articles 11, 12 and 14). The census does not evaluate these; it records privacy and accessibility links on the homepage. Source

Look up any website

Questions

How many of the top 100,000 websites block GPTBot?

7,305 of the 59,542 sites that serve a robots.txt block GPTBot for the whole site; 6,404 block ClaudeBot and 4,642 block ChatGPT-User; 2,957 block both ChatGPT-User and Claude-User; 14,912 name at least one AI crawler explicitly.

How many top sites publish a security.txt or an llms.txt?

7,061 of the 78,997 reachable sites (8.9%) publish a security.txt, 4,092 of them complete and current; 8,873 publish an llms.txt and 58 publish an A2A agent card.

Why is a well-known site missing or shown as unreachable?

Tranco ranks registrable domains, so subdomains fold into their parent. A domain is unreachable when neither the apex nor www answered over HTTPS within 8 seconds to one polite request from a US network; content-delivery hosts, API-only domains and sites that block automated requests are counted as the server answered.

Where does the ranking come from?

The Tranco list, a research ranking that combines several traffic-based lists; the census took the top 99,993 domains from the list pulled on 2026-09-29 and shows each site's rank on its page.

Can I reuse the data?

Yes. The findings, the CSV and every per-site JSON record are published under CC BY 4.0; cite Agent Exchange and link the report.

Method and limits

Population: Tranco top sites list, top 100,000 domains (list pulled 2026-09-29) a research ranking that combines several traffic lists; rank 1 is the most-visited domain. Types group domains by top-level domain (country codes upper-cased).. Fetch: one HTTPS request to the apex (then www.), following redirects, 8-second timeout, user agent agentexchange-census/1.0; then robots.txt, /.well-known/security.txt, /security.txt, /llms.txt and /.well-known/agent-card.json on the final host. "Reachable" means any HTTP response over HTTPS, including 403s from bot protection. Homepage links are detected from anchor targets and visible text, so a policy that exists but is not linked from the homepage is not counted. A robots.txt "blocked" verdict means the agent's own group (or the * group) disallows the whole site; "partly blocked" means some paths are disallowed. One snapshot per domain; slow sites and sites that block automated requests are counted as unreachable or reachable-with-error as the server answered. Per-domain records: lookup; downloads: CSV, NDJSON and JSON (CC BY 4.0).

Run one of these sites?

Instant AI visibility report for your website: $9 · $49 full report or $99 monthly monitoring · Hosted accessibility feedback and privacy request desk

Other censuses

AI crawler census (top 10,000 sites) · Government web census (.gov) · US bank web census · US college web census · US school district web census · US public library web census