Top 100,000 websites census · gap lists · census 2026-09-30 · JSON (top 50) · feed
Top-100k websites blocking GPTBot
7,305 of the 99,993 websites in the census list (7.3%; 9.2% of the 78,997 reachable over HTTPS), as fetched on 2026-09-30.
On this list
7,305
of 99,993 websites
Share of the list
7.3%
9.2% of reachable sites
Full list
1.00 USDC
x402, per file; first 50 rows free
What this gap means
robots.txt disallows GPTBot, OpenAI's training crawler, for the whole site.
Control: robots.txt verdict for GPTBot. Rule: blocks_gptbot over the census record (reachable sites only). Who buys this: AI-visibility and content-licensing vendors, publisher analytics, bot-management vendors.
The obligation it maps to
robots.txt (RFC 9309): The Robots Exclusion Protocol lets a site tell crawlers what to fetch; it is voluntary. The census records the verdict for GPTBot, ClaudeBot, ChatGPT-User, Claude-User, Perplexity-User and Googlebot and whether any AI crawler is named explicitly. Source
Free: the first 50 of 7,305 (by rank)
| Domain | TLD family | Rank | Census record |
|---|---|---|---|
| gstatic.com | com | 4 | HTTP 404 · HSTS no · security.txt none · privacy no · accessibility no · GPTBot blocked |
| instagram.com | com | 11 | HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| mail.ru | RU | 12 | HTTP 200 via account.mail.ru · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| twitter.com | com | 15 | HTTP 200 via x.com · HSTS yes · security.txt expired · privacy yes · accessibility yes · GPTBot blocked |
| dzen.ru | RU | 17 | HTTP 200 via sso.passport.yandex.ru · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| googlevideo.com | com | 19 | HTTP 404 · HSTS no · security.txt none · privacy no · accessibility no · GPTBot blocked |
| amazon.com | com | 26 | HTTP 200 · HSTS yes · security.txt incomplete · privacy no · accessibility no · GPTBot blocked |
| whatsapp.net | net | 32 | HTTP 200 via www.whatsapp.com · HSTS yes · security.txt complete · privacy yes · accessibility no · GPTBot blocked |
| doubleclick.net | net | 38 | HTTP 200 via marketingplatform.google.com · HSTS no · security.txt none · privacy yes · accessibility no · GPTBot blocked |
| x.com | com | 50 | HTTP 200 · HSTS yes · security.txt expired · privacy yes · accessibility yes · GPTBot blocked |
| pinterest.com | com | 53 | HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| tiktok.com | com | 54 | HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| whatsapp.com | com | 55 | HTTP 200 · HSTS yes · security.txt complete · privacy yes · accessibility no · GPTBot blocked |
| yahoo.com | com | 59 | HTTP 200 · HSTS yes · security.txt complete · privacy no · accessibility no · GPTBot blocked |
| spotify.com | com | 66 | HTTP 200 via open.spotify.com · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| wa.me | ME | 69 | HTTP 200 via api.whatsapp.com · HSTS yes · security.txt none · privacy yes · accessibility no · GPTBot blocked |
| myfritz.net | net | 70 | HTTP 200 · HSTS yes · security.txt complete · privacy no · accessibility no · GPTBot blocked |
| msn.com | com | 71 | HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| baidu.com | com | 81 | HTTP 200 · HSTS no · security.txt none · privacy no · accessibility no · GPTBot blocked |
| chatgpt.com | com | 82 | HTTP 403 · HSTS yes · security.txt incomplete · privacy no · accessibility no · GPTBot blocked |
| vimeo.com | com | 87 | HTTP 200 · HSTS yes · security.txt incomplete · privacy no · accessibility no · GPTBot blocked |
| yandex.ru | RU | 101 | HTTP 200 via sso.passport.yandex.ru · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| reddit.com | com | 109 | HTTP 200 · HSTS yes · security.txt complete · privacy no · accessibility no · GPTBot blocked |
| google-analytics.com | com | 119 | HTTP 200 via marketingplatform.google.com · HSTS no · security.txt none · privacy yes · accessibility no · GPTBot blocked |
| registrar-servers.com | com | 122 | HTTP 200 · HSTS no · security.txt none · privacy no · accessibility no · GPTBot blocked |
| snapchat.com | com | 130 | HTTP 200 · HSTS yes · security.txt incomplete · privacy yes · accessibility no · GPTBot blocked |
| gravatar.com | com | 154 | HTTP 200 · HSTS yes · security.txt none · privacy yes · accessibility no · GPTBot blocked |
| nytimes.com | com | 157 | HTTP 403 · HSTS yes · security.txt incomplete · privacy no · accessibility no · GPTBot blocked |
| flickr.com | com | 190 | HTTP 200 · HSTS no · security.txt incomplete · privacy yes · accessibility no · GPTBot blocked |
| soundcloud.com | com | 193 | HTTP 200 · HSTS yes · security.txt complete · privacy no · accessibility no · GPTBot blocked |
| medium.com | com | 203 | HTTP 403 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| cnn.com | com | 206 | HTTP 200 · HSTS no · security.txt none · privacy no · accessibility no · GPTBot blocked |
| ebay.com | com | 208 | HTTP 403 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| t-online.de | DE | 209 | HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| forbes.com | com | 221 | HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| bbc.com | com | 222 | HTTP 200 · HSTS yes · security.txt complete · privacy no · accessibility no · GPTBot blocked |
| sciencedirect.com | com | 226 | HTTP 403 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| bbc.co.uk | UK | 236 | HTTP 200 · HSTS yes · security.txt complete · privacy no · accessibility yes · GPTBot blocked |
| t.co | CO | 244 | HTTP 200 · HSTS yes · security.txt none · privacy yes · accessibility no · GPTBot blocked |
| canva.com | com | 247 | HTTP 403 · HSTS yes · security.txt incomplete · privacy no · accessibility no · GPTBot blocked |
| imdb.com | com | 269 | HTTP 202 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| launchpad.net | net | 284 | HTTP 200 · HSTS yes · security.txt none · privacy yes · accessibility no · GPTBot blocked |
| reuters.com | com | 292 | HTTP 401 · HSTS no · security.txt incomplete · privacy no · accessibility no · GPTBot blocked |
| nature.com | com | 304 | HTTP 200 · HSTS yes · security.txt none · privacy yes · accessibility yes · GPTBot blocked |
| alibaba.com | com | 315 | HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| skybridge.click | click | 337 | HTTP 200 · HSTS no · security.txt none · privacy no · accessibility no · GPTBot blocked |
| amazon.co.uk | UK | 338 | HTTP 200 · HSTS yes · security.txt incomplete · privacy no · accessibility no · GPTBot blocked |
| naver.com | com | 356 | HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
| bloomberg.com | com | 385 | HTTP 200 · HSTS yes · security.txt complete · privacy no · accessibility no · GPTBot blocked |
| amazon.de | DE | 400 | HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked |
The same rows as JSON: blocks-gptbot.top.json (CORS open, cached one day).
Get the full list: 7,305 rows, 1.00 USDC
x402 on Base, paid per file, no account or key. GET https://top100k.agentexchange.work/leads/blocks-gptbot.csv or GET https://top100k.agentexchange.work/leads/blocks-gptbot.json answers 402 Payment Required with the terms (scheme exact, USDC on Base, maxAmountRequired 1000000, payTo 0xc91cE6291eDC0713ec753BAFBA002506ffb2b95c); an x402 client signs the EIP-3009 authorization and resends the same request with X-PAYMENT; the file comes back with a PAYMENT-RESPONSE header. HEAD gets the same 402.
curl -sS -D - https://top100k.agentexchange.work/leads/blocks-gptbot.csv # 402 with the x402 terms; pay and resend with X-PAYMENT
Columns (19): domain, name, type, city, state, tranco_rank, reachable, http_only, https_status, final_host, hsts, security_txt, privacy_link, accessibility_link, robots_gptbot, robots_chatgpt_user, robots_claude_user, llms_txt, title. CSV: header row, every cell quoted, yes/no for booleans. JSON: { list, title, count, by_state, columns, rows[][] }. Sizes: CSV 1.2 MB, JSON 1.2 MB. The same facts are in the free CC BY 4.0 dumps: the price buys the filtered, state-sliceable file built from the 2026-09-30 census, not exclusive data. Settlement is final; no refunds. Prices in /.well-known/x402 equal the challenges. Terms · pricing.
Card checkout for this list is not open yet; the x402 route works now.
Monitor this list (free)
Atom feed: websites that entered or left this list according to the live re-checks (every 6 hours a rotating slice, each domain at most once per 14 days; each entry says which way it moved and which field changed). blocks-gptbot.top.json: the first 50 rows and the counts by state as JSON, CORS open. Fields watched: robots, robots.GPTBot.
Membership reflects the homepage and well-known files as fetched on 2026-09-30: a control that exists but was not found where the census looked counts as missing, and sites answering 403 to our one polite request count as reachable. Every domain links to its record with what to fix, a JSON twin, a badge and a changes feed. Data licence CC BY 4.0. Not legal advice.