Top 100,000 websites census · gap lists · census 2026-09-30 · JSON (top 50) · feed
Top-100k websites blocking both ChatGPT-User and Claude-User
2,957 of the 99,993 websites in the census list (3.0%; 3.7% of the 78,997 reachable over HTTPS), as fetched on 2026-09-30.
On this list
2,957
of 99,993 websites
Share of the list
3.0%
3.7% of reachable sites
Full list
1.00 USDC
x402, per file; first 50 rows free
What this gap means
robots.txt disallows both ChatGPT-User and Claude-User for the whole site, so neither assistant will open its pages for a user who asks.
Control: robots.txt verdicts for ChatGPT-User and Claude-User. Rule: blocks_assistants over the census record (reachable sites only). Who buys this: AI-visibility vendors, digital agencies and publisher analytics: sites closed to ChatGPT and Claude users.
The obligation it maps to
robots.txt (RFC 9309): The Robots Exclusion Protocol lets a site tell crawlers what to fetch; it is voluntary. The census records the verdict for GPTBot, ClaudeBot, ChatGPT-User, Claude-User, Perplexity-User and Googlebot and whether any AI crawler is named explicitly. Source
Free: the first 50 of 2,957 (by rank)
| Domain | TLD family | Rank | Census record |
|---|---|---|---|
| facebook.com | com | 3 | HTTP 200 · HSTS yes · security.txt complete · privacy no · accessibility no · GPTBot partly blocked |
| gstatic.com | com | 4 | HTTP 404 · HSTS no · security.txt none · privacy no · accessibility no · GPTBot blocked |
| instagram.com | com | 11 | HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| mail.ru | RU | 12 | HTTP 200 via account.mail.ru · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| fbcdn.net | net | 13 | HTTP 200 via www.facebook.com · HSTS yes · security.txt complete · privacy no · accessibility no · GPTBot partly blocked |
| twitter.com | com | 15 | HTTP 200 via x.com · HSTS yes · security.txt expired · privacy yes · accessibility yes · GPTBot blocked |
| dzen.ru | RU | 17 | HTTP 200 via sso.passport.yandex.ru · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| googlevideo.com | com | 19 | HTTP 404 · HSTS no · security.txt none · privacy no · accessibility no · GPTBot blocked |
| amazon.com | com | 26 | HTTP 200 · HSTS yes · security.txt incomplete · privacy no · accessibility no · GPTBot blocked |
| doubleclick.net | net | 38 | HTTP 200 via marketingplatform.google.com · HSTS no · security.txt none · privacy yes · accessibility no · GPTBot blocked |
| x.com | com | 50 | HTTP 200 · HSTS yes · security.txt expired · privacy yes · accessibility yes · GPTBot blocked |
| pinterest.com | com | 53 | HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| tiktok.com | com | 54 | HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| wa.me | ME | 69 | HTTP 200 via api.whatsapp.com · HSTS yes · security.txt none · privacy yes · accessibility no · GPTBot blocked |
| myfritz.net | net | 70 | HTTP 200 · HSTS yes · security.txt complete · privacy no · accessibility no · GPTBot blocked |
| msn.com | com | 71 | HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| baidu.com | com | 81 | HTTP 200 · HSTS no · security.txt none · privacy no · accessibility no · GPTBot blocked |
| chatgpt.com | com | 82 | HTTP 403 · HSTS yes · security.txt incomplete · privacy no · accessibility no · GPTBot blocked |
| yandex.ru | RU | 101 | HTTP 200 via sso.passport.yandex.ru · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| reddit.com | com | 109 | HTTP 200 · HSTS yes · security.txt complete · privacy no · accessibility no · GPTBot blocked |
| amazonvideo.com | com | 118 | HTTP 200 via www.primevideo.com · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot partly blocked |
| google-analytics.com | com | 119 | HTTP 200 via marketingplatform.google.com · HSTS no · security.txt none · privacy yes · accessibility no · GPTBot blocked |
| registrar-servers.com | com | 122 | HTTP 200 · HSTS no · security.txt none · privacy no · accessibility no · GPTBot blocked |
| nytimes.com | com | 157 | HTTP 403 · HSTS yes · security.txt incomplete · privacy no · accessibility no · GPTBot blocked |
| cnn.com | com | 206 | HTTP 200 · HSTS no · security.txt none · privacy no · accessibility no · GPTBot blocked |
| t.co | CO | 244 | HTTP 200 · HSTS yes · security.txt none · privacy yes · accessibility no · GPTBot blocked |
| imdb.com | com | 269 | HTTP 202 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| skybridge.click | click | 337 | HTTP 200 · HSTS no · security.txt none · privacy no · accessibility no · GPTBot blocked |
| amazon.co.uk | UK | 338 | HTTP 200 · HSTS yes · security.txt incomplete · privacy no · accessibility no · GPTBot blocked |
| naver.com | com | 356 | HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| bloomberg.com | com | 385 | HTTP 200 · HSTS yes · security.txt complete · privacy no · accessibility no · GPTBot blocked |
| amazon.de | DE | 400 | HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| amazonalexa.com | com | 450 | HTTP 200 via www.amazon.com · HSTS yes · security.txt incomplete · privacy yes · accessibility yes · GPTBot blocked |
| sohu.com | com | 481 | HTTP 200 · HSTS no · security.txt none · privacy no · accessibility no · GPTBot blocked |
| cnbc.com | com | 484 | HTTP 200 · HSTS no · security.txt none · privacy no · accessibility no · GPTBot blocked |
| quora.com | com | 496 | HTTP 403 · HSTS no · security.txt none · privacy no · accessibility no · GPTBot blocked |
| amazon.co.jp | JP | 497 | HTTP 200 · HSTS yes · security.txt incomplete · privacy no · accessibility no · GPTBot blocked |
| gosuslugi.ru | RU | 510 | HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| amzn.to | TO | 517 | HTTP 200 via www.amazon.com · HSTS yes · security.txt incomplete · privacy no · accessibility no · GPTBot blocked |
| amazon.co.za | ZA | 525 | HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| usatoday.com | com | 597 | HTTP 200 · HSTS yes · security.txt expired · privacy no · accessibility no · GPTBot blocked |
| threads.com | com | 609 | HTTP 200 · HSTS yes · security.txt complete · privacy no · accessibility no · GPTBot blocked |
| telegraph.co.uk | UK | 625 | HTTP 402 · HSTS no · security.txt complete · privacy no · accessibility no · GPTBot blocked |
| amazon.fr | FR | 626 | HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| amazon.ca | CA | 645 | HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| sharethrough.com | com | 648 | HTTP 200 · HSTS yes · security.txt none · privacy yes · accessibility no · GPTBot blocked |
| primevideo.com | com | 670 | HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot partly blocked |
| yelp.com | com | 676 | HTTP 403 · HSTS no · security.txt complete · privacy no · accessibility no · GPTBot blocked |
| amazon.in | IN | 690 | HTTP 200 · HSTS yes · security.txt incomplete · privacy no · accessibility no · GPTBot blocked |
| hihonorcloud.com | com | 691 | HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
The same rows as JSON: blocks-assistants.top.json (CORS open, cached one day).
Get the full list: 2,957 rows, 1.00 USDC
x402 on Base, paid per file, no account or key. GET https://top100k.agentexchange.work/leads/blocks-assistants.csv or GET https://top100k.agentexchange.work/leads/blocks-assistants.json answers 402 Payment Required with the terms (scheme exact, USDC on Base, maxAmountRequired 1000000, payTo 0xc91cE6291eDC0713ec753BAFBA002506ffb2b95c); an x402 client signs the EIP-3009 authorization and resends the same request with X-PAYMENT; the file comes back with a PAYMENT-RESPONSE header. HEAD gets the same 402.
curl -sS -D - https://top100k.agentexchange.work/leads/blocks-assistants.csv # 402 with the x402 terms; pay and resend with X-PAYMENT
Columns (19): domain, name, type, city, state, tranco_rank, reachable, http_only, https_status, final_host, hsts, security_txt, privacy_link, accessibility_link, robots_gptbot, robots_chatgpt_user, robots_claude_user, llms_txt, title. CSV: header row, every cell quoted, yes/no for booleans. JSON: { list, title, count, by_state, columns, rows[][] }. Sizes: CSV 446 KB, JSON 454 KB. The same facts are in the free CC BY 4.0 dumps: the price buys the filtered, state-sliceable file built from the 2026-09-30 census, not exclusive data. Settlement is final; no refunds. Prices in /.well-known/x402 equal the challenges. Terms · pricing.
Card checkout
Pay with a card through Stripe and get the file on the next page: Full list CSV, $29 · JSON, $29 · all 5 lists, $79. Agents pay less with x402 (see above).
Monitor this list (free)
Atom feed: websites that entered or left this list according to the live re-checks (every 6 hours a rotating slice, each domain at most once per 14 days; each entry says which way it moved and which field changed). blocks-assistants.top.json: the first 50 rows and the counts by state as JSON, CORS open. Fields watched: robots, robots.ChatGPT-User, robots.Claude-User.
Membership reflects the homepage and well-known files as fetched on 2026-09-30: a control that exists but was not found where the census looked counts as missing, and sites answering 403 to our one polite request count as reachable. Every domain links to its record with what to fix, a JSON twin, a badge and a changes feed. Data licence CC BY 4.0. Not legal advice.