Does robots.txt block AI agents?

No. It's a request, and not every agent follows it. ARC's HTTP block is a separate measurement from a robots.txt rule.

Does Shopify block ChatGPT by default?

Shopify's default robots.txt doesn't block AI crawlers from product pages; it disallows cart, checkout and account paths. ARC's Shopify counts on this page are from the dataset, not from that help page.

Does blocking crawlers remove me from AI shopping?

Not always. On Shopify, catalog data can still reach AI channels through Shopify Catalog. ARC does not measure catalog enrollment.

ARC · Answers

How should I set robots.txt for AI shopping agents?

robots.txt is advisory. It asks crawlers what to fetch but doesn't stop agents that ignore it. Naming a bot in its own group can switch off your default rules for that bot, so edit with care. Shopify's default doesn't block AI crawlers and adds a note pointing agents to official checkout routes.

What robots.txt can and can't do

robots.txt asks a crawler what to fetch. It does not stop an agent that ignores it, and it is not the same as an HTTP block. ARC records both, separately.

The common mistake: a named group replaces the * rules

Under RFC 9309, the most specific user-agent group wins. If you add a group that names one bot and you do not repeat the rules you still want, that bot no longer uses the * group. Edit with care. This page does not publish a robots.txt file to paste.

Shopify's default and its agent note

Shopify's help says the default file does not block AI crawlers from product pages, disallows cart, checkout and account paths, and adds a note pointing agents to official checkout routes. That is Shopify's documentation. The rates below are what ARC counted.

ARC open dataset snapshot 2026-10-12. Fields blocks_ai_crawlers and blocks_ai_answer_agents on store_answers. rails_checks.csv in this snapshot has no robots columns, so these figures are not taken from that file.

robots.txt disallows at least one tracked AI crawler: 160 of 3,166 (5.1%)Measured331 finished checks could not read robots.txt and stay outside this rate. Unknown is not a block. A missing row is not a no.

robots.txt disallows an answer agent: 142 of 3,894 (3.6%)MeasuredThis source does not split unreadable robots.txt from the rate. Unknown is not a block, and a missing row is not a no. Answer agents are the tokens that fetch a page for a shopper. A rule in a file is not proof the store turned the agent away.

Shopify: robots.txt disallows a tracked crawler: 18 of 1,624 (1.1%)Measured2 finished checks could not read robots.txt and stay outside this rate. Unknown is not a block. A missing row is not a no.

Shopify: robots.txt disallows an answer agent: 12 of 1,624 (0.7%)Measured2 finished checks could not read robots.txt and stay outside this rate. Unknown is not a block. A missing row is not a no.

Crawlers vs shopping agents

blocks_ai_crawlers is yes when robots.txt disallows at least one tracked crawler (answer, search or training). blocks_ai_answer_agents is the narrower set that fetches a page for a shopper. Training crawlers and shopping agents are different decisions.

A safe starting point

Leave the platform default unless you know which group you are changing. If you add a named group, copy the rules you still want into that group. Then recheck the store. A rule is not a verdict that every agent was turned away.

Check the result on ARC

Your store page shows the robots finding next to the HTTP block. Which stores block AI agents lists the answer-agent rules in the dataset. Should I block AI agents? is the decision, not the file format.

Frequently asked questions

How should I set robots.txt for AI shopping agents?: robots.txt is advisory. It asks crawlers what to fetch but doesn't stop agents that ignore it. Naming a bot in its own group can switch off your default rules for that bot, so edit with care. Shopify's default doesn't block AI crawlers and adds a note pointing agents to official checkout routes.
Does robots.txt block AI agents?: No. It's a request, and not every agent follows it. ARC's HTTP block is a separate measurement from a robots.txt rule.
Does Shopify block ChatGPT by default?: Shopify's default robots.txt doesn't block AI crawlers from product pages; it disallows cart, checkout and account paths. ARC's Shopify counts on this page are from the dataset, not from that help page.
Does blocking crawlers remove me from AI shopping?: Not always. On Shopify, catalog data can still reach AI channels through Shopify Catalog. ARC does not measure catalog enrollment.

On ARC

Cite this: ARC, arcreport.ai, accessed 2026-10-13, CC-BY-4.0. Protocol figures: ARC saved checks, accessed 2026-10-13 (13 October 2026). Index browser figures: ARC open dataset snapshot 2026-10-12. Platform figures: ARC saved checks, accessed 2026-10-13.

Every figure is labelled Measured, Inferred or Not measured. Not measured is not zero. A missing result is not a mark against a store. ARC stops before payment.