Top 100,000 websites census

Top 100,000 websites census · gap lists · census 2026-09-30 · JSON (top 50) · feed

Top-100k websites blocking GPTBot

7,305 of the 99,993 websites in the census list (7.3%; 9.2% of the 78,997 reachable over HTTPS), as fetched on 2026-09-30.

On this list

7,305

of 99,993 websites

Share of the list

7.3%

9.2% of reachable sites

Full list

1.00 USDC

x402, per file; first 50 rows free

What this gap means

robots.txt disallows GPTBot, OpenAI's training crawler, for the whole site.

Control: robots.txt verdict for GPTBot. Rule: blocks_gptbot over the census record (reachable sites only). Who buys this: AI-visibility and content-licensing vendors, publisher analytics, bot-management vendors.

The obligation it maps to

robots.txt (RFC 9309): The Robots Exclusion Protocol lets a site tell crawlers what to fetch; it is voluntary. The census records the verdict for GPTBot, ClaudeBot, ChatGPT-User, Claude-User, Perplexity-User and Googlebot and whether any AI crawler is named explicitly. Source

Free: the first 50 of 7,305 (by rank)

DomainTLD familyRankCensus record
gstatic.comcom4HTTP 404 · HSTS no · security.txt none · privacy no · accessibility no · GPTBot blocked
instagram.comcom11HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked
mail.ruRU12HTTP 200 via account.mail.ru · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked
twitter.comcom15HTTP 200 via x.com · HSTS yes · security.txt expired · privacy yes · accessibility yes · GPTBot blocked
dzen.ruRU17HTTP 200 via sso.passport.yandex.ru · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked
googlevideo.comcom19HTTP 404 · HSTS no · security.txt none · privacy no · accessibility no · GPTBot blocked
amazon.comcom26HTTP 200 · HSTS yes · security.txt incomplete · privacy no · accessibility no · GPTBot blocked
whatsapp.netnet32HTTP 200 via www.whatsapp.com · HSTS yes · security.txt complete · privacy yes · accessibility no · GPTBot blocked
doubleclick.netnet38HTTP 200 via marketingplatform.google.com · HSTS no · security.txt none · privacy yes · accessibility no · GPTBot blocked
x.comcom50HTTP 200 · HSTS yes · security.txt expired · privacy yes · accessibility yes · GPTBot blocked
pinterest.comcom53HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked
tiktok.comcom54HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked
whatsapp.comcom55HTTP 200 · HSTS yes · security.txt complete · privacy yes · accessibility no · GPTBot blocked
yahoo.comcom59HTTP 200 · HSTS yes · security.txt complete · privacy no · accessibility no · GPTBot blocked
spotify.comcom66HTTP 200 via open.spotify.com · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked
wa.meME69HTTP 200 via api.whatsapp.com · HSTS yes · security.txt none · privacy yes · accessibility no · GPTBot blocked
myfritz.netnet70HTTP 200 · HSTS yes · security.txt complete · privacy no · accessibility no · GPTBot blocked
msn.comcom71HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked
baidu.comcom81HTTP 200 · HSTS no · security.txt none · privacy no · accessibility no · GPTBot blocked
chatgpt.comcom82HTTP 403 · HSTS yes · security.txt incomplete · privacy no · accessibility no · GPTBot blocked
vimeo.comcom87HTTP 200 · HSTS yes · security.txt incomplete · privacy no · accessibility no · GPTBot blocked
yandex.ruRU101HTTP 200 via sso.passport.yandex.ru · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked
reddit.comcom109HTTP 200 · HSTS yes · security.txt complete · privacy no · accessibility no · GPTBot blocked
google-analytics.comcom119HTTP 200 via marketingplatform.google.com · HSTS no · security.txt none · privacy yes · accessibility no · GPTBot blocked
registrar-servers.comcom122HTTP 200 · HSTS no · security.txt none · privacy no · accessibility no · GPTBot blocked
snapchat.comcom130HTTP 200 · HSTS yes · security.txt incomplete · privacy yes · accessibility no · GPTBot blocked
gravatar.comcom154HTTP 200 · HSTS yes · security.txt none · privacy yes · accessibility no · GPTBot blocked
nytimes.comcom157HTTP 403 · HSTS yes · security.txt incomplete · privacy no · accessibility no · GPTBot blocked
flickr.comcom190HTTP 200 · HSTS no · security.txt incomplete · privacy yes · accessibility no · GPTBot blocked
soundcloud.comcom193HTTP 200 · HSTS yes · security.txt complete · privacy no · accessibility no · GPTBot blocked
medium.comcom203HTTP 403 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked
cnn.comcom206HTTP 200 · HSTS no · security.txt none · privacy no · accessibility no · GPTBot blocked
ebay.comcom208HTTP 403 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked
t-online.deDE209HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked
forbes.comcom221HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked
bbc.comcom222HTTP 200 · HSTS yes · security.txt complete · privacy no · accessibility no · GPTBot blocked
sciencedirect.comcom226HTTP 403 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked
bbc.co.ukUK236HTTP 200 · HSTS yes · security.txt complete · privacy no · accessibility yes · GPTBot blocked
t.coCO244HTTP 200 · HSTS yes · security.txt none · privacy yes · accessibility no · GPTBot blocked
canva.comcom247HTTP 403 · HSTS yes · security.txt incomplete · privacy no · accessibility no · GPTBot blocked
imdb.comcom269HTTP 202 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked
launchpad.netnet284HTTP 200 · HSTS yes · security.txt none · privacy yes · accessibility no · GPTBot blocked
reuters.comcom292HTTP 401 · HSTS no · security.txt incomplete · privacy no · accessibility no · GPTBot blocked
nature.comcom304HTTP 200 · HSTS yes · security.txt none · privacy yes · accessibility yes · GPTBot blocked
alibaba.comcom315HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked
skybridge.clickclick337HTTP 200 · HSTS no · security.txt none · privacy no · accessibility no · GPTBot blocked
amazon.co.ukUK338HTTP 200 · HSTS yes · security.txt incomplete · privacy no · accessibility no · GPTBot blocked
naver.comcom356HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked
bloomberg.comcom385HTTP 200 · HSTS yes · security.txt complete · privacy no · accessibility no · GPTBot blocked
amazon.deDE400HTTP 200 · HSTS yes · security.txt none · privacy no · accessibility no · GPTBot blocked

The same rows as JSON: blocks-gptbot.top.json (CORS open, cached one day).

Get the full list: 7,305 rows, 1.00 USDC

x402 on Base, paid per file, no account or key. GET https://top100k.agentexchange.work/leads/blocks-gptbot.csv or GET https://top100k.agentexchange.work/leads/blocks-gptbot.json answers 402 Payment Required with the terms (scheme exact, USDC on Base, maxAmountRequired 1000000, payTo 0xc91cE6291eDC0713ec753BAFBA002506ffb2b95c); an x402 client signs the EIP-3009 authorization and resends the same request with X-PAYMENT; the file comes back with a PAYMENT-RESPONSE header. HEAD gets the same 402.

curl -sS -D - https://top100k.agentexchange.work/leads/blocks-gptbot.csv   # 402 with the x402 terms; pay and resend with X-PAYMENT

Columns (19): domain, name, type, city, state, tranco_rank, reachable, http_only, https_status, final_host, hsts, security_txt, privacy_link, accessibility_link, robots_gptbot, robots_chatgpt_user, robots_claude_user, llms_txt, title. CSV: header row, every cell quoted, yes/no for booleans. JSON: { list, title, count, by_state, columns, rows[][] }. Sizes: CSV 1.2 MB, JSON 1.2 MB. The same facts are in the free CC BY 4.0 dumps: the price buys the filtered, state-sliceable file built from the 2026-09-30 census, not exclusive data. Settlement is final; no refunds. Prices in /.well-known/x402 equal the challenges. Terms · pricing.

Card checkout for this list is not open yet; the x402 route works now.

Monitor this list (free)

Atom feed: websites that entered or left this list according to the live re-checks (every 6 hours a rotating slice, each domain at most once per 14 days; each entry says which way it moved and which field changed). blocks-gptbot.top.json: the first 50 rows and the counts by state as JSON, CORS open. Fields watched: robots, robots.GPTBot.

Membership reflects the homepage and well-known files as fetched on 2026-09-30: a control that exists but was not found where the census looked counts as missing, and sites answering 403 to our one polite request count as reachable. Every domain links to its record with what to fix, a JSON twin, a badge and a changes feed. Data licence CC BY 4.0. Not legal advice.