Census fetched 2026-09-30 · population: Tranco top sites list, top 100,000 domains (list pulled 2026-09-29) · datasets (CC BY 4.0) · live changes
The top 100,000 websites census, September 2026: what the web's most-visited domains publish for security researchers and AI agents
The 99,993 highest-ranked domains in the Tranco list (list pulled 2026-09-29) were fetched once over HTTPS on 2026-09-30, with robots.txt, security.txt (both locations), llms.txt and the A2A agent card. 78,997 answered with a website. Lists and lookup pages are ordered by Tranco rank.
Reachable over HTTPS
79.0%
78,997 of 99,993 websites
security.txt
8.9%
7,061 of reachable sites; 4,092 complete and current
Accessibility statement linked
6.7%
5,293 homepages
Privacy policy linked
30.3%
23,928 homepages
Block GPTBot
7,305
4,642 block ChatGPT-User; 6,404 ClaudeBot
llms.txt
8,873
58 publish an A2A agent card
What we found
- security.txt: 7,061 of 78,997 reachable sites (8.9%) publish one; 4,092 are complete and current (a Contact line and an unexpired Expires line, RFC 9116); 918 have expired; 742 serve it only at /security.txt instead of /.well-known/security.txt.
- Accessibility: 6.7% of reachable sites link an accessibility statement from the homepage.
- Privacy: 30.3% link a privacy policy from the homepage.
- HTTPS: 78,997 of 99,993 domains (79.0%) answered over HTTPS; 3,800 answer only over plain HTTP; 46.0% of reachable sites send HSTS; 6,773 homepages redirect to a different domain.
- AI crawlers and assistants: 59,542 sites serve a robots.txt; 7,305 block GPTBot, 6,404 ClaudeBot and 4,642 ChatGPT-User; 2,241 disallow every bot; 2,957 block both ChatGPT-User and Claude-User; 3,756 block GPTBot while leaving OAI-SearchBot allowed; 14,912 name an AI crawler explicitly; 1,303 carry a Content-Signal line.
- Machine-readable surfaces: 8,873 publish an llms.txt and 58 an A2A agent card; 26,647 homepages carry schema.org structured data.
Live re-checks
Since 2026-10-01 this host has re-checked 10,943 websites (a rotating slice every 6 hours, each domain at most once per 14 days; last run 2026-10-25 18:43 UTC) and recorded 3,004 changes against the census records. What changed · Atom feed · live.json.
By tld family
| TLD family | In list | Reachable | security.txt | valid | privacy link | accessibility link | HSTS | blocks GPTBot | blocks ChatGPT-User | llms.txt | agent card |
|---|---|---|---|---|---|---|---|---|---|---|---|
| com | 45,411 | 36,850 (81.1%) | 3,067 (8.3%) | 1,609 | 35.1% | 6.2% | 46.7% | 3,445 | 2,235 | 5,147 | 25 |
| other | 26,852 | 21,414 (79.7%) | 2,269 (10.6%) | 1,440 | 23.6% | 7.0% | 46.3% | 1,910 | 1,199 | 1,959 | 19 |
| net | 6,272 | 3,581 (57.1%) | 260 (7.3%) | 153 | 28.3% | 2.9% | 35.5% | 353 | 262 | 263 | 1 |
| org | 4,512 | 4,001 (88.7%) | 318 (7.9%) | 220 | 39.9% | 9.3% | 45.7% | 365 | 214 | 295 | 2 |
| RU | 3,578 | 2,601 (72.7%) | 82 (3.2%) | 20 | 16.9% | 0.6% | 30.4% | 184 | 123 | 157 | 0 |
| DE | 2,087 | 1,808 (86.6%) | 407 (22.5%) | 281 | 11.4% | 3.5% | 62.7% | 302 | 147 | 138 | 0 |
| JP | 1,954 | 1,587 (81.2%) | 35 (2.2%) | 22 | 49.0% | 6.8% | 35.9% | 143 | 104 | 81 | 0 |
| IO | 1,868 | 1,352 (72.4%) | 137 (10.1%) | 79 | 41.9% | 2.4% | 53.3% | 65 | 53 | 324 | 8 |
| BR | 1,464 | 1,260 (86.1%) | 71 (5.6%) | 50 | 4.4% | 1.3% | 46.9% | 57 | 35 | 155 | 2 |
| CN | 1,429 | 727 (50.9%) | 24 (3.3%) | 12 | 7.7% | 1.4% | 29.0% | 28 | 22 | 27 | 0 |
| UK | 1,398 | 1,140 (81.5%) | 169 (14.8%) | 100 | 44.4% | 29.4% | 61.9% | 161 | 67 | 87 | 0 |
| IN | 1,288 | 979 (76.0%) | 28 (2.9%) | 17 | 27.3% | 5.1% | 49.3% | 49 | 35 | 138 | 0 |
| FR | 1,008 | 902 (89.5%) | 131 (14.5%) | 58 | 11.4% | 25.9% | 56.8% | 117 | 70 | 42 | 0 |
| IT | 872 | 795 (91.2%) | 63 (7.9%) | 31 | 46.2% | 19.6% | 51.4% | 126 | 76 | 60 | 1 |
One page per tld family with its domains: https://top100k.agentexchange.work/by-tld-family.
Named lists
Largest websites without a security.txt (first 60 of 300)
gstatic.com · googleapis.com · instagram.com · mail.ru · dzen.ru · linkedin.com · googlevideo.com · googletagmanager.com · office.com · live.com · azure.com · appsflyersdk.com · bing.com · fastly.net · googleusercontent.com · doubleclick.net · cloud.microsoft · skype.com · icloud.com · pinterest.com · tiktok.com · goo.gl · cloudflare.net · googledomains.com · endpoints.news · spotify.com · wa.me · msn.com · qq.com · zoom.us · nginx.org · baidu.com · cloudflare-dns.com · tiktokv.com · f5.com · samsung.com · yandex.ru · t.me · blogspot.com · yuga.com · b-cdn.net · nic.ru · office365.com · europa.eu · amazonvideo.com · google-analytics.com · okcdn.ru · outlook.com · registrar-servers.com · app-analytics-services.com · github.io · app-measurement.com · triplinkintl.com · reg.ru · intuit.com · unity3d.com · nih.gov · kaspersky.com · forms.gle · aliyuncs.com
Sites with an expired security.txt (first 60 of 300)
amazonaws.com · microsoft.com · twitter.com · x.com · windows.com · booking.com · dell.com · ivi.ru · usatoday.com · ikea.com · kick.com · bild.de · welt.de · life360.com · princeton.edu · jimdo.com · xing.com · wise.com · crazygames.com · gofundme.com · sectigo.com · divar.ir · abc.net.au · deezer.com · va.gov · timeanddate.com · dynamics.com · mparticle.com · amazonaws.com.cn · coinbase.com · surfshark.com · mos.ru · rockstargames.com · franceinfo.fr · affirm.com · capitalone.com · comodoca.com · bt.com · powerapps.com · yr.no · tamu.edu · zalando.de · watchguard.com · n-able.com · telenet.be · razorpay.com · trouble-free.net · azurecontainerapps.io · dinahosting.com · powerplatform.com · smartsheet.com · urbandictionary.com · sprinklr.com · faceit.com · aws.com · unl.edu · smallpdf.com · chewy.com · rte.ie · wandera.com
Sites publishing an llms.txt (first 60 of 1,000)
cloudflare.com · fastly.net · digicert.com · wordpress.org · adobe.com · workers.dev · opera.com · sentry.io · samsung.com · wordpress.com · b-cdn.net · dropbox.com · intuit.com · unity3d.com · kaspersky.com · gravatar.com · webex.com · one.one · paypal.com · shopify.com · base.org · hosting24.com · macromedia.com · rubiconproject.com · ubuntu.com · vungle.com · taboola.com · pages.dev · comcast.net · stripe.com · sourceforge.net · roku.com · hubspot.com · linktr.ee · hcaptcha.com · wixsite.com · forter.com · amplitude.com · weather.com · dell.com · etsy.com · hostinger.com · name.com · bilibili.com · slack.com · checkpoint.com · selectel.ru · gitlab.com · zendesk.com · nvidia.com · datadoghq.com · plesk.com · calendly.com · dailymotion.com · skyhigh.cloud · intel.com · paloaltonetworks.com · wp.com · okta.com · bitdefender.net
Sites publishing an A2A agent card (58)
forter.com · monday.com · razorpay.com · opensea.io · glance.com · vidyard.com · alerts.in.ua · activehosted.com · activecampaign.com · zahid.host · modelcontextprotocol.io · builtwith.com · rhombus.com · rhombussystems.com · telnyx.com · agentclientprotocol.com · techaro.lol · shields.io · dash0.com · tavily.com · openarchieven.nl · metadata.io · voiceflow.com · pinecone.io · gymbeam.cz · yippy.com · gogs.io · any.do · shapermint.com · leadpages.net · leadpages.com · gymbeam.sk · lobehub.com · gymbeam.hu · buser.com.br · attio.com · rzp.io · pydantic.dev · gymbeam.ro · getname.pl · freefiremania.com.br · surrealdb.com · chatboxai.app · frognews.bg · bcdtravel.com · gymbeam.pl · agentskills.io · gymbeam.gr · xrpl.org · shrenz.com · loi.com.uy · gymbeam.ua · skillsmp.com · sportra.app · quicknode.com · gymbeam.it · bazel.build · arkive.org
Sites telling ChatGPT-User and Claude-User not to open pages for users (first 60 of 300)
facebook.com · gstatic.com · instagram.com · mail.ru · fbcdn.net · twitter.com · dzen.ru · googlevideo.com · amazon.com · doubleclick.net · x.com · pinterest.com · tiktok.com · wa.me · myfritz.net · msn.com · baidu.com · chatgpt.com · yandex.ru · reddit.com · amazonvideo.com · google-analytics.com · registrar-servers.com · nytimes.com · cnn.com · t.co · imdb.com · skybridge.click · amazon.co.uk · naver.com · bloomberg.com · amazon.de · amazonalexa.com · sohu.com · cnbc.com · quora.com · amazon.co.jp · gosuslugi.ru · amzn.to · amazon.co.za · usatoday.com · threads.com · telegraph.co.uk · amazon.fr · amazon.ca · sharethrough.com · primevideo.com · yelp.com · amazon.in · hihonorcloud.com · dailymail.com · dailymail.co.uk · fb.com · amazon.es · pinimg.com · cnet.com · amazon.it · quantserve.com · creativecdn.com · crwdcntrl.net
Sites blocking GPTBot (OpenAI's training crawler) (first 60 of 300)
gstatic.com · instagram.com · mail.ru · twitter.com · dzen.ru · googlevideo.com · amazon.com · whatsapp.net · doubleclick.net · x.com · pinterest.com · tiktok.com · whatsapp.com · yahoo.com · spotify.com · wa.me · myfritz.net · msn.com · baidu.com · chatgpt.com · vimeo.com · yandex.ru · reddit.com · google-analytics.com · registrar-servers.com · snapchat.com · gravatar.com · nytimes.com · flickr.com · soundcloud.com · medium.com · cnn.com · ebay.com · t-online.de · forbes.com · bbc.com · sciencedirect.com · bbc.co.uk · t.co · canva.com · imdb.com · launchpad.net · reuters.com · nature.com · alibaba.com · skybridge.click · amazon.co.uk · naver.com · bloomberg.com · amazon.de · espn.com · globo.com · go.com · claude.ai · amazonalexa.com · yahoo.co.jp · aol.com · pixabay.com · goodreads.com · sohu.com
Homepages that redirect to another domain (first 60 of 300)
amazonaws.com → aws.amazon.com · fbcdn.net → www.facebook.com · twitter.com → x.com · dzen.ru → sso.passport.yandex.ru · office.com → m365.cloud.microsoft · hicloudcam.com → www.ezviz.com · azure.com → azure.microsoft.com · whatsapp.net → www.whatsapp.com · fastly.net → www.fastly.com · doubleclick.net → marketingplatform.google.com · youtu.be → www.youtube.com · skype.com → teams.live.com · googledomains.com → domains.google · googlesyndication.com → www.google.com · wa.me → api.whatsapp.com · zoom.us → www.zoom.com · workers.dev → www.cloudflare.com · cloudflare-dns.com → one.one.one.one · nginx.com → www.f5.com · windows.com → www.microsoft.com · t.me → telegram.org · blogspot.com → www.blogger.com · actify.nl → www.zilverenkruis.nl · b-cdn.net → bunny.net · bit.ly → bitly.com · amazonvideo.com → www.primevideo.com · google-analytics.com → marketingplatform.google.com · outlook.com → outlook.live.com · github.io → pages.github.com · unity3d.com → unity.com · discord.gg → discord.com · xiaomi.com → www.mi.com · lencr.org → letsencrypt.org · adobe.io → developer.adobe.com · adtrafficquality.google → www.google.com · one.one → www.one.com · vkuserphoto.ru → ps.userapi.com · hosting24.com → www.hostinger.com · macromedia.com → www.adobe.com · paytmfirstgames.com → www.firstgames.in · telekom.net → www.telekom.com · rubiconproject.com → www.magnite.com · vungle.com → liftoff.ai · pages.dev → www.cloudflare.com · comcast.net → www.xfinity.com · meraki.com → www.cisco.com · gmail.com → accounts.google.com · wixsite.com → www.wix.com · crashlytics.com → firebase.google.com · mzstatic.com → www.apple.com · telegram.me → telegram.org · googleblog.com → blog.google · aliyun.com → www.alibabacloud.com · mozilla.com → www.firefox.com · onelink.me → www.appsflyer.com · kwai-pro.com → hotvideo.kwai.com · timeweb.ru → timeweb.com · aliexpress.com → www.aliexpress.us · skyhigh.cloud → www.skyhighsecurity.com · kwaipros.com → hotvideo.kwai.com
Sites with a Content-Signal line in robots.txt (first 60 of 200)
cloudflare.com · sentry.io · workers.dev · hosting24.com · zoom.us · pages.dev · amplitude.com · linktr.ee · launchpad.net · hostinger.com · forter.com · nvidia.com · bitdefender.net · trendmicro.com · vercel.app · oxylabs.io · unesco.org · patreon.com · intercom.io · zoom.com · netlify.app · theatlantic.com · supabase.co · corriere.it · usercentrics.eu · bbb.org · mydramalist.com · psu.edu · twilio.com · otomoto.pl · dagbladet.no · segment.io · therapservices.net · wpastra.com · twkan.com · imgsrc.ru · scan-manga.com · intercom.com · bitdefender.com · pewresearch.org · moodle.org · hover.com · livechatinc.com · vinted.fr · segment.com · ninjarmm.com · yotpo.com · metamask.io · laravel.com · onrender.com · customer.io · cookiebot.com · sendbird.com · gameanalytics.com · coinmarketcap.com · railway.app · redis.io · cookieyes.com · padlet.com · vinted.com
Largest websites without an accessibility link on the homepage (first 60 of 300)
google.com · cloudflare.com · facebook.com · gstatic.com · googleapis.com · amazonaws.com · youtube.com · instagram.com · mail.ru · fbcdn.net · dzen.ru · linkedin.com · googlevideo.com · googletagmanager.com · live.com · hicloudcam.com · amazon.com · azure.com · github.com · wikipedia.org · appsflyersdk.com · whatsapp.net · bing.com · fastly.net · googleusercontent.com · doubleclick.net · netflix.com · gandi.net · youtu.be · digicert.com · wordpress.org · skype.com · icloud.com · pinterest.com · tiktok.com · whatsapp.com · goo.gl · cloudflare.net · googledomains.com · yahoo.com · googlesyndication.com · roblox.com · endpoints.news · spotify.com · adobe.com · wa.me · myfritz.net · msn.com · ntp.org · qq.com · zoom.us · nginx.org · workers.dev · baidu.com · chatgpt.com · cloudflare-dns.com · tiktokv.com · vimeo.com · openai.com · opera.com
Gap lists for vendors: who is missing which control
Each list is the websites whose 2026-09-30 record lacks one control, with counts by state, the first 50 rows free, an Atom feed of who enters or leaves it, and the full CSV or JSON for 1.00 USDC (x402). All gap lists.
| List | Websites | Of the list | Who buys this |
|---|---|---|---|
| Top-100k websites with an expired security.txt | 918 | 0.9% | Security teams and vulnerability-disclosure vendors: a lapsed file is a specific, verifiable conversation opener. |
| Top-100k websites blocking both ChatGPT-User and Claude-User | 2,957 | 3.0% | AI-visibility vendors, digital agencies and publisher analytics: sites closed to ChatGPT and Claude users. |
| Top-100k websites blocking GPTBot | 7,305 | 7.3% | AI-visibility and content-licensing vendors, publisher analytics, bot-management vendors. |
| Top-100k domains answering only over plain HTTP | 3,800 | 3.8% | MSPs, hosting and web agencies: sites still without working HTTPS. |
| Top-100k homepages that redirect to another domain | 6,773 | 6.8% | Web agencies, domain and brand-protection vendors, security teams checking for stale redirects. |
What websites are expected to publish
- robots.txt (RFC 9309): The Robots Exclusion Protocol lets a site tell crawlers what to fetch; it is voluntary. The census records the verdict for GPTBot, ClaudeBot, ChatGPT-User, Claude-User, Perplexity-User and Googlebot and whether any AI crawler is named explicitly. Source
- security.txt (RFC 9116): A file at /.well-known/security.txt with Contact and Expires fields tells researchers where to report vulnerabilities. The census marks a file complete when both fields are present and the Expires date has not passed. Source
- llms.txt: A proposed Markdown file at /llms.txt that summarizes a site for AI assistants and links its most useful pages. The census counts a file that returns 200, is not HTML and starts with a Markdown heading. Source
- A2A agent card: The Agent2Agent protocol publishes a JSON card at /.well-known/agent-card.json describing an agent's skills and endpoint. The census counts a card that parses as JSON and carries a name or skills. Source
- Content Signals: A robots.txt extension (Content-Signal line) that states whether content may be used for search, AI input and AI training. The census counts robots.txt files that carry the line. Source
- EU Digital Services Act: point of contact and terms: Providers of intermediary services offered in the EU must designate a single point of contact for users and authorities and publish clear terms and conditions (Regulation (EU) 2022/2065, Articles 11, 12 and 14). The census does not evaluate these; it records privacy and accessibility links on the homepage. Source
Look up any website
Questions
How many of the top 100,000 websites block GPTBot?
7,305 of the 59,542 sites that serve a robots.txt block GPTBot for the whole site; 6,404 block ClaudeBot and 4,642 block ChatGPT-User; 2,957 block both ChatGPT-User and Claude-User; 14,912 name at least one AI crawler explicitly.
How many top sites publish a security.txt or an llms.txt?
7,061 of the 78,997 reachable sites (8.9%) publish a security.txt, 4,092 of them complete and current; 8,873 publish an llms.txt and 58 publish an A2A agent card.
Why is a well-known site missing or shown as unreachable?
Tranco ranks registrable domains, so subdomains fold into their parent. A domain is unreachable when neither the apex nor www answered over HTTPS within 8 seconds to one polite request from a US network; content-delivery hosts, API-only domains and sites that block automated requests are counted as the server answered.
Where does the ranking come from?
The Tranco list, a research ranking that combines several traffic-based lists; the census took the top 99,993 domains from the list pulled on 2026-09-29 and shows each site's rank on its page.
Can I reuse the data?
Yes. The findings, the CSV and every per-site JSON record are published under CC BY 4.0; cite Agent Exchange and link the report.
Method and limits
Population: Tranco top sites list, top 100,000 domains (list pulled 2026-09-29) a research ranking that combines several traffic lists; rank 1 is the most-visited domain. Types group domains by top-level domain (country codes upper-cased).. Fetch: one HTTPS request to the apex (then www.), following redirects, 8-second timeout, user agent agentexchange-census/1.0; then robots.txt, /.well-known/security.txt, /security.txt, /llms.txt and /.well-known/agent-card.json on the final host. "Reachable" means any HTTP response over HTTPS, including 403s from bot protection. Homepage links are detected from anchor targets and visible text, so a policy that exists but is not linked from the homepage is not counted. A robots.txt "blocked" verdict means the agent's own group (or the * group) disallows the whole site; "partly blocked" means some paths are disallowed. One snapshot per domain; slow sites and sites that block automated requests are counted as unreachable or reachable-with-error as the server answered. Live re-checks: every 6 hours this host re-fetches the homepage, robots.txt, security.txt and llms.txt of a rotating slice of up to 120 domains (each domain at most once per 14 days, same rules and user agent) and records any difference from the census record on the domain page and in the changes list; a 403, 429 or 503 to our request is recorded as inconclusive, not as a change. Per-domain records: lookup; downloads: CSV, NDJSON and JSON (CC BY 4.0).
Run one of these sites?
Instant AI visibility report for your website: $9 · $49 full report or $99 monthly monitoring · Hosted accessibility feedback and privacy request desk
Other censuses
US government web census (.gov) · World government web census · US bank web census · US credit union web census · US college web census · US school district web census · World university web census · Every Agent Exchange property (network map) · Agent-readiness grade and fix guides · Compliance Desk: hosted request processes · MCP registry integrity report · Agent Economy Index (daily x402 and MCP census) · Agent Exchange directory (x402 store)