The state of agent access, today
Every number below is computed from the latest scan of 1,015brands and regenerates with each scan — nothing is hand-written. Each stat has a stable anchor and a ready-to-paste citation. Definitions: "blocking" counts robots.txt policy blocks andWAF restrictions; "fully open" means no agent blocked or restricted.
Scan date: 2026-06-20 · Machine-readable: /insights.md
418 of 1015 scanned brands neither block nor restrict any tracked agent.
Includes robots.txt policy blocks and WAF/CDN-level restrictions.
4 robots.txt policy blocks + 498 WAF restrictions across 1015 brands.
39 of 71 food-beverage brands allow every tracked agent.
Presence of /llms.txt detected in the latest scan.
38 distinct brands changed at least one signal in the last 7 days.
Blocking rate per agent
| Agent | Company | Policy blocks | WAF restricted | % blocking | |
|---|---|---|---|---|---|
| GPTBot | OpenAI | 26 | 449 | 46.8% | |
| ChatGPT-User | OpenAI | 11 | 435 | 43.9% | |
| ClaudeBot | Anthropic | 29 | 439 | 46.1% | |
| Claude-Web | Anthropic | 14 | 0 | 1.4% | |
| PerplexityBot | Perplexity | 14 | 425 | 43.3% | |
| Google-Extended | 24 | 421 | 43.8% | ||
| Amazonbot | Amazon | 30 | 441 | 46.4% | |
| Bingbot | Microsoft | 4 | 498 | 49.5% | |
| CCBot | Common Crawl | 37 | 428 | 45.8% |
Openness by category
| Category | Brands | Fully open | % fully open | Avg agents blocked |
|---|---|---|---|---|
| fashion | 155 | 69 | 44.5% | 3.5 |
| dtc | 99 | 45 | 45.5% | 3.2 |
| electronics | 89 | 47 | 52.8% | 2.3 |
| beauty | 80 | 28 | 35% | 4.2 |
| home | 77 | 25 | 32.5% | 4.2 |
| luxury | 73 | 26 | 35.6% | 4.1 |
| food-beverage | 71 | 39 | 54.9% | 2.7 |
| general | 68 | 19 | 27.9% | 5.1 |
| sports | 68 | 27 | 39.7% | 3.8 |
| health | 60 | 32 | 53.3% | 3.1 |
| grocery | 48 | 14 | 29.2% | 4.9 |
| kids | 45 | 15 | 33.3% | 4.2 |
| automotive | 42 | 11 | 26.2% | 4.7 |
| pet | 40 | 21 | 52.5% | 2.7 |
Platform breakdown
| Platform | Brands | % of index |
|---|---|---|
| magento | 430 | 42.4% |
| custom | 400 | 39.4% |
| salesforce | 102 | 10% |
| woocommerce | 45 | 4.4% |
| nextjs | 19 | 1.9% |
| shopify | 13 | 1.3% |
| unknown | 6 | 0.6% |
Citation format: "stat — ARC Report, 2026-06-20, arcreport.ai/insights#anchor". Data is CC BY 4.0 (download). Methods: /methodology.